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EXECUTIVE  SUMMARY 


tin  purpose  of  this  project  was  to  investigate  state  of  the  art 
■if  MIS  software  effectiveness  evaluation  criteria  and  assess  the 
desirability  of  embarking  on  an  expanded  MIS  software  effectiveness 
project  in  the  future. 

Those  objectives  were  achieved  in  four  phases:  first,  a 
management  information  metrics  was  developed.  Then,  the  current 
uvuiiabl'  measurement  techniques  were  analyzed.  The  third  phase 
included  an  assessment  of  the  extent  to  which  available  techniques 
can  evaluate  management  information  attributes  (summarized  in  Table 
7-4) ,  an.l  the  outcome  of  this  phase  was  identification  of  research 
needs.  final iv,  the  fourth  phase  recommends  guidelines  for  a  larger 
MIS  ei feet iveness  project. 

i'he  measurement  techniques  analyzed  were  classified  into 
four  groups:  economic,  behavioral,  other  and  management  science. 
After  examining  about  seventeen  different  techniques,  the  following 
major  find inns  were  arrived  at: 

-  there  is  no  single  satisfactory  approach  that  can  measure 
and  evaluate  MIS  software  effectiveness  from  the  user's 
perspective,  mainly  because  of  deficiencies  in  establishing 
the  theoretical  metric  and  shortcomings  of  the  measuring 
devices, 

-  development  of  surrogate  measures  to  evaluate  effectiveness 
is  a  promising  direction  to  pursue,  " 
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-  no  methodology  exists  for  evaluating  the  effectiveness  of 
the  system  based  upon  measurement  of  the  system  attributes. 

Research  needs  were  defined  in  the  form  of  "research  clusters,' 
where  each  cluster  represents  a  number  of  related  research  topics. 
Four  such  clusters  were  identified  -  measurement ,  effectiveness , 
design  j>hase  and  future;  trends  cluster. 

Recommended  guidelines  for  the  MIS  effectiveness  project 
include  four  "modules"  -  a  project  module  for  each  research  cluster 
It  is  further  recommended  that  an  MIS  software  effectiveness  project 
be  designed  and  implemented  following  the  modular  structure.  The 
project  implementation  policy  could  use  a  parallel  approach  - 
implementing  ail  four  modules  at  the  same  time,  or  a  sequential 
approach  -  one  module  at  a  time,  where  the  priorities  are: 

-  Measurement  module 

-  Effectiveness  module 

-  Design  Phase  module 

-  Future  Trends  module 

The  sequential  approach  is  recommended.  Furthermore,  it  is 
recommended  that  the  first  two  modules  be  implemented  concurrently. 
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1.  INTRODUCTION 


1.1  Purpose 

The  United  States  Army  Institute  for  Research  in  Management 
Informat  in  and  Computer  Science  (AIRMICS)  has  initiated  an  exploratory 
research  in  the  area  of  management  information  systems  (MIS) 
software  effectiveness.  The  major  objectives  of  this  research  are 
to  establish  MLS  software-effectiveness  evaluation  state-of-the-art 
and  assess  the  desirability  of  embarking  on  an  expanded  MIS 
effectiveness  project  in  the  future. 

1_._2  Mot  ivat  ion 

The  motivation  for  performing  this  research  stems  from  the 
recognition  that  there  is  a  clear  need  in  the  Army  to  identify, 
develop  and  implement  methods  to  evaluate  MIS  software  effectiveness, 
not  only  related  to  current  operations,  but  also  with  a  view  towards 
future  developments,  namely,  computer  technologies,  distributed 
systems  and  modern  communication  interfaces.  Furthermore,  this 
research  relates  to  some  USACSC  command  objectives  and  is  further 
supported  by  the  findings  and  recommendations  of  the  Second  Software 
Life  Cycle  Management  Workshop. 

1_._1 _ Met hods  and  Scope 

The  amount  of  effort  allocated  to  this  study  was  restricted  to 
sixty  man  days  between  the  period  15  June  and  10  September  1979. 

The  major  part  of  the  effort  was  devoted  to  a  comprehensive 
literature  search  and  analysis  and  reporting  of  the  findings,  and 
discussions  with  AIRMICS  personnel.  The  balance  of  the  effort  was 
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al  local  ed  to  visits  to  U.S.  Army  installations  -  Fort  McPherson  in 
Atlanta,  Ceorgia,  and  Fort  Hood,  Texas,  where  discussions  were 


held  w i tli  the  personnel  there.  The  objective  of  these  discussions 
was  to  obtain  a  feel  of  the  user's  perspective  of  MIS  effectiveness. 

1 . 4  Organization  of  the  Report 

Chapter  2  describes  the  environment  within  which  the  current 
military  MIS  (STAMMIS)  is  used.  Chapter  3  summarizes  the  literature 
search  effort.  Chapter  4  combines  a  discussion  of  the  nature  of 
the  decision  making  process  with  the  history  of  MTS  to  present  some 
current  problems  in  MIS.  Chapter  5  discusses  some  basic  concepts 
related  to  MIS  evaluation  and  presents  a  management  information 
metrics.  Chapter  6  presents  the  state-of-the-art  in  MIS  effectiveness 
evaluation  techniques.  Chapter  7  assesses  those  techniques  in 
relation  to  the  management  information  metrics.  Chapter  8  discusses 
various  topics  related  to  MIS  effectiveness.  Finally,  Chapter  9 
identities  research  needs,  and  Chapter  10  recommends  and  gives 
guidelines  for  an  MIS  effectiveness  project.  A  bibliography  list  is 
inc  Luded  a  1  so . 
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2. 


THE  SYSTEM  ENVIRONMENT 


The  system  environment  related  to  this  research  is  a 
mill t i command  system  composed  of  DA,  MACOMS,  CSC,  proponent 
agencies,  DPI  and  more.  Each  component  of  the  system  is  either 
information  producer  or  information  user  or  both.  It  is  obvious 
that  t lie  information  requirements  will  vary  between  the  various 
components  of  the  system  and  within  them.  However,  one  should 
realize  that  information  is  the  life  blood  system  that  connects 
the  different  information  users,  i.e.,  the  decision  makers.  The 
information  flow  is  handled  by  STAMMIS  -  Standard  Army  Multicommand 
Management  Information  System. 

Some  criticism  has  recently  been  leveled  at  this  system, 
especially  from  lower  echelons,  where  the  major  concern  is  that 
the  system  has  a  "stove-pipe"  feature,  i.e.,  lower  echelons  feed 
t lie  pipe  with  information  that  services  the  needs  of  a  higher  level 
of  the  organization,  but  do-not  provide  any  benefit  to  the  "feeding 
echelons . " 

An  example  of  this  concern  is  seen  in  an  extract  from  the 
Fort  Hood  IMS  report  (March  1979),  as  follows: 

Most  STAMMIS  are  considered  to  be  inadequate  management 
tools  at  installation  level  because  they  are: 

1.  Designed  to  support  functional  management  at  DA  level, 
not  at  installation  level. 

2.  Vertically  structured  to  support  a  very  narrow  segment 
of  the  functional  responsibility  associated  with 
installation  management. 

J.  Independent  and  have  little  communication  or  interface 
between  systems. 
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h .  Collectors  and  reporters  of  data  without  comparison 
to  a  previously  established  standard  or  reference 
point;  thov  tend  to  be  conduits  for  transmitting 
great  quantities  of  raw  data. 

Predominantly  operational  systems  rather  than 
management  systems. 

Despite  the  obvious  weaknesses  of  the  existing  information 
sv  items  for  installation  management,  they  are  not  changeable 
through  local  action  and  must  therefore  be  used  as  presently 

observed . 

This  lengthy  quote  does  not  mean  endorsing  the  deficiencies 
identit  led.  However,  it  does  present  perceived  dissatisfaction 

of  a  major  user  from  the  current  MIS,  and  indicates  the  need  to  J 

i 

I  j 

identify  available  techniques  of  evaluating  MIS  software  effectiveness,  j 

;  I 

and  identify  research  needs  in  this  area,  from  the  user's  point  of  < 

view.  The  balance  of  this  report  addresses  this  problem. 
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3.  LITERATURE  SURVEY 


1.1  Int  roilnc  t  ion 

Tlu*  literature  survey  presented  here  does  not  follow  the 
regular  approach  of  a  brief  description  of  each  one  of  the  relevant 
references.  The  reason  is  that  in  this  study,  a  major  part  is 
establishing  the  "state-of-the-art"  in  MIS  software  effectiveness, 
and  therefore  it  was  found  to  be  more  beneficial  to  describe  the 
relevant  references  along  with  the  specific  topic  investigated. 

This  way  a  better  relationship  between  topics  and  references  can 
he  established.  Therefore,  this  chapter  will  concentrate  on  the 
literature  search  effort  that  was  done,  and  will  give  a  general 
frame  of  reference  to  the  various  literature  sources. 

The  literature  search  effort  included  the  following  activities: 

1.  Computer ized  literature  search  of  five  "data  bases,"  as 

follows:  NT LS ,  MGMT  CONTENTS,  COMPENDEX,  INSPEC,  ABl/ INFORM 

l.  !>I)C  computerized  literature  search 

3.  Manual  library  search. 

This  effort  yielded  about  300  references.  Initial  screening 
reduced  this  number  to  about  50,  which  were  closely  reviewed.  All 
the  references  that  were  reviewed  are  listed  in  the  bibliography 
list.  Out  of  this  list,  about  30  references  are  cited  in  the  report 
for  specifics,  and  the  rest  were  used  as  a  general  background 
material . 

3. 2  Ref ere nee s  Grouping 

In  order  to  introduce  some  structure  into  the  reference  list, 
grouping  was  performed,  and  the  references  were  divided  into  five 
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as  tell  uws  : 

1.  MIS  1, valuation  -  includes  those  references  that  arc  mainly 
concerned  with  various  aspects  of  the  evaluation  process. 

2.  Future  Freuds 

i.  MIS  -  references  that  deal  with  general  MIS  issues,  and  that 
v."  re  useful  in  the  investigation  of  this  study.  This  is  not 
a  comprehensive  list,  as  the  area  is  "flooded"  with 
publications,  however,  most  of  them  were  found  to  be  irrelevant 

to  tin-  .valuation  issue. 

4.  Cost  Benefit 

5.  M i ScuJ  i uneous  -  different  reference,  less  specific,  that 
were  found  to  be  related  to  the  current  study. 

The  classification  of  the  references  Is  given  in  Table  3-1.  In 
each  the  references  are  arranged  in  descending  order  of  their 

year  u:  pu!  iication.  Each  reference  has  a  one  or  two  word  description 
of  its  !:.  'i-r  'fra  f  .  and  also  a  classification  whether  it*s  a  book  (B), 
paper ,  report .  article  etc.  (P) ,  or  a  Ph.D.  dissertation  (D)i 
Auu  i  t  iuiu-i  in'  irmat ;  ui  about  each  reference  can  be  obtained  by  inspecting 
th.  b ibl iograpiiv  list,  or  by  reading  the  text  for  those  references 
that  >.  re  cited.  (Note:  in  case  of  two  authors  or  more,  only  the  first 
one  is  listed  in  fable  3-1). 

1 u  general,  the  references  that  proved  to  be  most  useful  to 
til  is  research  are:  Dumas  (1978),  Liggon  (1978),  Keen  (1975)  King  and 
ClcaLand  (1973),  Lucas  (1975),  Mason  (1973),  Parden  (1978),  Murdick 
and  Ross  (19/3),  Davis  (1974),  U.S.  Army  Report  (Fort  Hood  1979), 

Anthony  (1963). 
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Table  3-1.  Literature  Search  -  Survey 


Evaluation  (D)  Simulation  (P)  Simulation 


4.  EVOLUTION  OF  MIS 


4.1  Introduction 

The  concept  ot  Management  Information  Systems  is  one  of  those 
ambiguous  terms  that  means  different  things  to  different  people,  and 
apparently  could  be  compared  to  the  ambiguity  of  the  term  "system 
analysis."  l'he  literature  is  full  of  various  definitions  of  MIS. 

For  the  purpose  of  this  study,  the  definition  given  by  Davis  (1974) 
seems  to  he  most  appropriate. 

A  manager, icr;  t  information  system,  or  MIS,  is  an  information  system 
that,  in  addition  to  providing  all  necessary  transaction  processing 
tor  an  organization,  provides  information  and  processing  support 
for  management  and  decision  functions.  The  idea  of  such  an 
information  system  preceded  the  advent  of  the  computers,  but 
computers  made  the  idea  feasible. 

The  above  definition  implies  that  MIS  is  not  merely  a  data 
pnn  ess i up  a.  t  ivitv,  but  an  activity  that  has  to  supply  information  in 
order  to  support  the  managerial  decision  process.  Furthermore,  the 
concept  ot  information  supporting  management  decisions  existed  long 
be’ on  the  computer  era.  In  a  way,  it  might  be  claimed  that  the  essence 
ot  the  decision  making  process  lias  not  changed  that  much  over  the  years, 
fhe  big  change  has  been  in  the  tools  and  understanding  of  the  process. 

lo  gain  further  understanding  of  the  current  status  of  MIS,  first 
the  decision  making  process  is  reviewed,  followed  by  the  history  of  MIS, 
culminating  with  a  discussion  of  some  of  the  problems  in  this  area. 

4.2  The  Decision  Making  Process 

No  attempt  is  going  to  be  made  here  to  present  a  comprehensive 
examination  of  the  decision  making  process.  However,  in  order  to  be 
nbLe  to  evaluate  MIS  effectiveness,  one  has  to  have  an  appreciation  of 
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tin.'  process  MIS  is  supposed  to  serve  -  especially  within  a  military 
environment.  Following  is  a  brief  description  of  some  of  the  current 
approaches  and  theories  describing  this  process,  with  an  emphasis 
on  the  information  needs. 

Simon  (1965)  has  described  three  major  components  of  the  decision 
making  process,  as  follows: 

-  Intel  1 igcnce;  involves  searching  the  environment  or  becoming  aware 
of  the  situation  that  requires  a  decision. 

-  Design;  the  decision  maker  has  to  enumerate  and  evaluate  the 
alternatives  available. 

-  Cho  ice ;  the  decision  maker  selects  from  the  alternatives  delineated 
during  design. 

Lt  might  be  useful  to  add  another  step  to  Simon's  model  -  implementation, 
the  process  of  carrying  out  the  decision. 

Information  systems  have  the  potential  for  supporting  all  parts  of 
the  decision  making  process  outlined  above. 

Simon's  approach  to  the  decision  making  process  is  technical  in 
nature.  The  more  conceptual  approaches  are  summarized  very  well  in  Keen 
and  Morton  (1978).  Five  views  of  the  decision  making  process  are 
presented  as  follows: 

-  The  economic,  rational  concept;  Decision  makers  are  all  knowing  and 
able  to  evaluate  alJL  alternatives.  They  are  dissatisfied  with  any 
solution  but  the  best.  This  approach  represents  the  classical 
normative  theory  of  decision  making.  (Described  in  early  works  of 
Cyert,  Simon  and  Trow) 


-  Tin*  satisficing  concept;  This  is  a  process  oriented  view,  where 
decision  makers  are  considered  to  be  rational,  although  cognitive 
limits  lead  to  a  "bounded  rationality,"  making  a  decision  maker 
desire  to  get  a  good  enough  answer,  not  the  best  possible  one. 

(Simon's  approach) 

-  Hie  organizational  procedure  concept;  This  approach  highlights 
the  organizational  structure  formal  and  informal  mechanisms 

tor  communication  and  coordination,  and  the  standard  operating 

t 

procedures  by  which  decision  making  is  systematized  (Cyert  and 
March's  "A  Behavioral  Theory  of  the  Firm"  is  the  most  complete 
statement  of  this  approach.) 

_  The  political  concept;  The  participants  in  the  decision  making 
process  are  regarded  as  actors  with  parts  to  play.  Coalitions 
or  organizational  subgroups  are  formed,  and  decisions  are 
frequently  dominated  by  bargaining  and  conflict,  resulting  with 
only  minor  changes  in  the  status  quo.  (A  good  definition  of 
this  concept  is  given  in  Allison  G.T.:  Essence  of  Decisions,  1971) 

-  The  individual  differences  concept;  The  claim  here  is  that  an 
individual's  personality  and  style  strongly  determine  his  or  her 
choices  and  behavior,  which  is  very  much  determined  by  the  manner 
in  which  an  individual  processes  information.  (See  for  example, 
Schroder,  Driver  and  Steufert:  "Human  Information  Processing," 
1967) 

Obviously,  those  approaches  to  the  decision  making  process  are  not 
mutually  exclusive.  They  vary  from  the  entirely  normative  to  the 
entirely  descriptive.  The  real  problem  is  not  to  develop  one  grand  all 
inclusive  theory,  but  to  be  aware  of  the  many  paths  through  it.  Certain 
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systems  can  be  described  better  by  one  approach,  and  others  by  a 


different  one. 

Military  systems  may  be  better  described,  but  not  necessarily 
better  understood,  by  the  organizational  procedure  concept.  The 
management  process  associated  with  this  concept  can  be  viewed  as 
composed  of  five  functions: 

-  planning 

-  organizing 

-  staffing 

-  directing 

-  controlling 

Within  this  context  of  activities,  it  is  possible  to  identify 
three  types  of  management  decisions  that  have  to  be  supported  by 
information  systems: 

-  technical 

-  tactical 

-  strategic 

A  further  insight  into  management  decisions  is  given  by 
Drucker  (1977)  who  identified  three  categories: 

-  Operational  Decisions:  not  really  decisions,  because  they 
involve  no  risk  and  are  programmable 

-  Managerial  Decisions:  primarily  deal  with  the  allocation  of 
resources  including  people,  for  which  there  is  no  "right" 
answers,  and  therefore  they  involve  risk. 

-  Kntrepreneural  Decisions:  have  no  right  answer;  one  seeks  to 
take  the  right  risk  to  innovate  and  change  the  trend  rather 
than  follow  or  anticipate  it. 
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Those  types  of  decisions  require  two  types  of  information  systems, 
defined  as : 

-  programmable,  or  operations  systems 

-  nonprogrammable,  or  management  information  systems 
both  support  the  function  of  "management,"  however,  management 
information  systems  support  decision  making  by  managers  -  a  much 
more  difficult  task.  Parden  (1978)  describes  this  distinction  on  a 
continuum  oi  organ tzat ional  styles  (after  Desler) ,  as  follows: 


People  who  bring 
skills  to  work 


People  who  bring 
knowledge  to  work 


Programmable 

decisions 


a 


o 


Nonprogrammable 

decisions 


i'pt  rat  ion  information 
s  v  s  t.  ems 


Management  information 
systems 


Figure  4-1.  Continuum  of  Organizational  Styles 

Tin  -  continuum  should  be  observed  when  the  effectiveness  of  MIS 
is  going,  to  be  considered  much  so  because  the  cost  effectiveness 
of  operations  systems  can  readily  be  determined,  while  the  value  of 
information  supporting  management  decision  making  is  always  vague. 

To  conclude  this  discussion  of  the  decision  making  process,  it 
is  worthwhile  to  consider  it  from  the  decision  maker  point  of  view 
and  not  the  decision  making .  According  to  Mason  and  Mitroff  (1978) 
the  decision  maker  is 


s 
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....  one  person  of  a  certain  psychological  type  who  faces  a 
problem  within  some  organizational  context  for  which  he  needs 
evidence  to  arrive  at  a  solution,  and  that  evidence  is  made 
available  to  him  through  some  mode  of  presentation. 

Following  the  concepts-  developed  by  Churchman,  they  present  five 
archetypal  ways  of  modeling  and  generating  evidence  for  any  problem. 
These  archetypal  ways  are  labelled  "inquiring  systems"  (IS)  defined 
as  follows: 

-  Lockean  fS:  -  are  experimental,  consensual  systems 

-  Leibnitizian  IS:  -  are  formal,  symbolic  systems 

-  Kamtian  IS:  -  multi  model,  synthetic  systems 

-  Hegelian  (Dialechical)  IS:  -  conflictual,  synthetic  systems 

-  Singer  tan  -  Chur chinan ian  IS:  -  involve  continual  learning  and 
adaptation  through  feedback. 

ft  should  be  noted  that  Singerian  IS  are  best  suited  for  studying 
all  of  the  rest  TS,  although  most  of  the  MIS  systems  are  considered 
from  the  standpoint  of  Leibnitian  and  Lockean  inquiry. 

-i._  1 _ Historical  Review 

One  should  realize  at  the  outset  that  information  was  an  important 
facet  ot  any  organization  survival  even  before  the  computer  arrived  on 
the  organization  scene.  The  computer  opened  up  new  horizons  for  using 
information  in  support  of  the  organization's  activities.  The  use  of 
the  computer  was  an  evolutionary  process,  where  four  causes  can  be 
identified  as  associated  with  this  gross  phenomena  (Walsh,  1978) 

-  development  of  application  portfolio  (early  1960's) 

-  building  of  an  EDP  Organization  (middle  and  late 

1960’s) 

-  building  an  EDP  management  control  system  (early 
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dev  I  oping  'i  user  aware.ncss  (middle  and  late  1970's) 

Those  lour  causes  were  paralled  by  four  basic  stages  of  EDP 

growth,  as  follows: 

-  cost  reducing  accounting  applications  (early  1960 's) 

-  proliferation  ot  applications  in  all  functional  areas  (middle 
and  late  1960's) 

-  emphasize  on  control  (early  1970 's) 

-  Oat  a  base  applications  (middle  and  late  1970's). 

Drwing  Stage  1  of  the  evolution  process,  users  found  the  computer 
to  be  a  terriitc  tool  i.n  helping  to  reduce  manpower  costs,  especially 
in  urea;;  life  payroll,  accounting  and  finance.  The  emphasis  was  on 
data  pro;  easing. 

Stage  IT,  which  came  about  in  the  late  1960's,  represents  the 
t  ranu i t  ion  i r  cm  data  processing  to  MIS.  To  the  original  effort  were 
add<*d  budgeting,  :oncasting,  inventory  control  and  others.  The  major 
role  ot  the  computer  as  a  data  processor  started  shifting.  This 
peiio.i  saw  the  expansion  of  computer  facilities,  with  more  equipment 
and  staff  added,  and  more  sophis t leaf ed  software  developed,  demanding 
budge  t  all  ocat  ion^-  . 

Stage  111  was  a  period  of  consolidation.  The  l  mcern  about  M1S/EDP 
expenditures  has  grown,  and  the  general  feeling  was  that  it  was  more 
economical  to  do  many  jobs  in  a  few  large  computers  at  one  central 
site  than  in  many  small  computers  at  a  number  of  local  sites.  During 
this  stage,  controls  and  standards  were  instituted  and  enforced. 

Stage  IV,  is,  in  a  way,  the  future  state  of  being  of  the  MIS/EDP 
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systems.  The  user  is  recognized  as  a  full  partner  in  MIS  activity 
known  factors,  which  were  not  recognized  before,  are  making  themselves 
felt,  such  as  minicomputers,  data  bases  and  distributed  systems. 

Most  organizations,  including  the  military  system,  are  in 
Stage  Ilf.  Large  data  centers  are  handling  the  needs  of  many  users, 
where  the  major  concern  is  to  make  the  operation  more  efficient. 

4 . 4 _ The  Problem  Environment 

From  the  discussion  so  far,  three  major  facts  related  to  MIS, 
emerge  ns  follows: 

1 .  Computer  based  MIS  grew  out  of  data  processing  into 
information  systems  supporting  management  decision  making 
process . 

2.  The  decision  making  process  varies  according  to  the  type  of 
decision  that  has  to  be  made,  and  this  process  centers 
around  the  human  element. 

1.  Tn  MIS,  the  focus  was  initially  on  procedures  and  Instruments, 
and  only  recently  shifted  to  the  persons  who  utilize  it. 

Data  processing  systems  are  very  technical  in  nature,  MIS  is  much 
more  "human."  In  a  way,  data  processing  is  one  element  of  MIS. 
Therefore,  evaluating  both  systems  cannot  be  performed  using  the  same 
methodology.  Furthermore,  when  evaluating  those  two  systems,  one  is 
concerned  with  two  different  things,  namely:  efficiency  for  data 
processing,  effectiveness  for  MIS.  Before  proceeding^ a  further 
clarification  of  those  terms  is  required. 

According  to  Keen  and  Morton  (1978);  efficiency  means  performing 
a  given  task  as  well  as  possible  in  relation  to  some  predefined 
performance  criterion.  Ef f ectiveness  involves  identifying  what  should 
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be  done  and  ensuring  that  the  chosen  criterion  is  the  relevant  one. 

Thus,  effectiveness  is  setting  the  criterion,  efficiency  is  comparing 
with  the  criterion.  Therefore,  a  computer  center  may  be  very 
efficient  in  the  process  of  generating  management  reports  that  nobody 
uses,  i.e.,  the  center  is  very  ef f ic ient  in  pursuit  of  ineffective 
goals . 

It  is  possible  now  to  appreciate  the  problem  environment  of 
MIS.  During  the  "data  processing"  period,  efficiency  was  the 
proper  approach.  Since  data  processing  is  more  technical  in  nature, 
defining  the  criterion,  and  measuring  it  was  much  easier  to  do. 

Various  metrics  were  defined,  such  as  reliability  metrics,  flexibility 
metrics,  resource  metrics,  etc.  (Gilb  1977).  As  the  evolution  from 
"data"  to  "information”  took  place,  the  same  metrics  were  retained, 
for  measuring  information,  i.e.,  the  efficiency  approach  is  used, 
whereas  what  is  needed  is  effectiveness.  Definitely,  the  same 
criteria  used  fcr  data  systems  do  not  apply  anymore,  as  MIS  is  more 
user  oriented  and  less  "technical"  oriented. 

Wiat  is  needed  is  a  "users"  point  of  view  to  evaluate  the  MIS 
and  not  a  "computer"  point  of  view.  After  all,  if  a  system  is  not 
u sed ,  it  cannot  be  considered  a  success,  even  if  it  functions  well 
technically . 

One  explanation  to  the  current  state  of  affairs  is  that  historically 
computer  scientists  dealt  with  data  processing  systems,  and  "moved" 
with  it  to  MIS,  where  an  additional  skill  in  management  systems  and 
behavioral  sciences  is  required. 

The  MIS  evaluation  problem  attracted  management  attention  in 
recent  years  because  of  the  increased  investment  in  MIS  software,  that 


h.i.s  to  he  justified  on  the  basis  of  c lie  benefits  obtained.  During 
t(ie  i.ite  idSO’s  and  early  1960's,  most  of  the  investment  In  computers 
was  in  hardware,  and  only  a  small  fraction  of  it  went  into  software. 
Since  then,  the  percentage  that  goes  into  MIS  software  has  steadily 
increased,  to  the  point  that  this  has  to  be  justified  like  any 
other  investment.  That  prompted  an  increased  interest  in  the 


problem  of  MIS  sol tware  effectiveness. 
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5.  MLS  SOKTWARK  EVALUATION:  BASIC  CONCEPTS 

3  .1  1  at  rotiuc  t  ion 

This  chapter  serves  as  a  prelude  to  the  following  sections  of 
this  report.  Before  any  further  discussion  of  MIS  software 
effectiveness  can  take  place,  it  might  be  worthwhile  to  consider 
some  basic  concepts  associated  with  this  process.  Thus,  the 
difference  between  data  and  information  is  first  defined,  leading 
into  discussion  of  the  difference  between  evaluation  and  measurement, 
which  are  the  basic  concepts  required  to  assess  effectiveness. 

Finally,  information  attributes  are  discussed,  culminating  with  a 
management  information  metrics. 

3.2  On  Data  and  Information 

Following  Murdick  and  Ross  (1975'f  definition,  "information 
is  tfie  behavior  initiating  stimuli  between  sender  and  receiver. 
Information  is  in  the  form  of  signs  that  are  coded  representation 
of  data." 

Data  is  information  if  it  somehow  modifies  the  decision  maker's 
image.  Data  may  be  considered  to  be  some  kind  of  recorded 
observations,  that  are  not  currently  affecting  behavior.  Data  mav 
become  information  if  behavior  becomes  affected.  Thus,  if  a  stack 
of  reports  is  delivered  to  a  decision  maker  and  he  throws  up  his 
hands  in  disgust,  the  data  in  the  reports  have  not  become  information. 
Information  may  be  defined  then  as  "data  in  use",  or  "information 
is  the  net  value  obtained  from  the  process  of  matching  the  elements 
of  a  present  problem  with  appropriate  elements  of  data"  (McDonoug! 
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Il»<>  1) .  Tliis  is  .1  very  basic  concept  when  considering  MIS,  since 
data  must  he  delivered  to  decision  makers  as  information  to  be 
acted  upon.  Too  often  M LS  are  treated  as  data  systems  rather 
titan  .is  into rm.a_t  ion  systems. 

5.  i  hva  i  tta  t  i_tm  and  Measurement 

Measurement,  in  its  most  general  sense,  is  basically  the 
process  of  ascribing  a  numerical  value  to  an  object  or  quality. 
Typically,  it  is  a  two  stage  process;  the  first  step  is  setting 
the  measuring  concept  -  the  theoretical  metric,  like  the  idea  of 
tin*  volt .  The  second  step  is  finding  a  practical  measuring 
device,  like  the  voltmeter. 

Effective  use  of  metrics  is  well  recognized  in  business  and 
engineering.  Absence  of  metrics  can  lead  to  lack  of  control  over 
systems,  and  final Iv  to  failure.  The  fact  that  some  system 
attribute  has  never  been  measured  before,  or  cannot  be  measured 
directly  or  accurately,  should  not  discourage  the  attempt  to 
construct  some  measuring  device  so  that  certain  control  can  be 
maintained  over  the  system. 

The  metrical  content  of  an  attribute  is  a  measure  in  a  common 
frame  of  reference. 

Evaluation,  especially  in  the  context  of  MIS,  is  a  much 
broader  term,  as  it  implies  value  judgement,  in  addition  to 
measuring.  Following  Keen  (1975),  evaluation  implies  the 
comparison  between  the  output  of  the  system  (actual  or  predicted) 
and  some  criterion  of  success.  Furthermore,  success,  when  dealing 
with  MIS,  implies  consideration  of  the  environment  in  which 
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Tlu'  il  i  I  I Vrenrp  between  eva  I  u.it  i  on  and  measuroment  may  be 
looked  upon  also  in  tin-  following  way.  Measurement  assigns 
numbers .  evaluation  assigns  vaj  ue .  Thus,  the  process  of  evaluation 
does  not  necessarily  involve  the  use  of  numbers. 

It  is  worthwhile  at  this  point  to  review  some  of  the  problems 
associated  with  the  measurement  process.  In  the  scientific  world, 
consistency  of  the  scale  is  one  of  the  major  concerns.  This  is 
even  more  so  for  measurement  within  an  organizational  environment. 

The  comparison  element,  imbedded  in  the  measurement  process  is  also 
a  matter  of  interest ,  as  it  is  not  certain  that  with  the  same 
perfect  scale  two  different  observers  will  find  the  same  results, 
final iy,  especially  for  measurements  in  an  organizational  context, 
there  is  the  problem  of  t lie  influence  of  the  observer  on  the  object 
to  be  measured.  Those  problems  should  he  kept  in  mind  when  dealing 
later  wi tii  measuring  MIS  software  effectiveness. 

From  both  scientific  and  convenience  points  of  view,  it  is 
highly  desirable  to  assign  numerals  to  objects  which  are  to  be 
compared.  However,  in  MIS,  such  assignment  is  most  of  the  times 
not  easv  to  perform.  Therefore,  the  approach  of  "surrogate 
measures"  has  been  developed,  where,  when  it  is  difficult  to  assign 
a  measure  to  an  object  or  quality,  another  available  measurement, 
or  set  of  measurements  is  used  to  represent  the  impossible  one. 

5 . 4  Management  Information  Attributes 

One  of  the  objectives  of  this  research  is  to  assess,  among  other 
things,  MIS  software  effectiveness  techniques  -  implying  considering 
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•i  user's  point  ol  view.  The  basis  lor  this  kind  of  investigation 
is  an  understandi ng  of  the  management  information  attributes.  It 
should  he  mentioned  at  the  outset  that  this  is  an  area  that 
requires  further  research.  The  discussion  here  summarizes  some  of 
the  current  thinking  in  this  direction,  as  represented  in  the 
literature.  The  measurement  aspects  of  those  attributes  is 
discussed  in  a  latter  chapter. 

One  of  the  frst  major  works  recognizing  MIS  within  the 
managerial  framework  and  defining  management  information  attributes 
is  that  of  Anthonv  (1963).  Three  levels  of  management  are 
identified,  as  follows: 

-  Strategic  Planning  (top  management) 

Policies,  objectives,  resources  etc. 

-  Management  Control  (middle  management) 

Ktfective  and  efficient  utilization  of  resources  in  the 

accomplishment  of  the  organization's  objectives. 

-  Operational.  Control  (operating  management) 

Carrying  out  specific  task  effectively  and  efficiently 

The  information  requirement  for  each  management  level,  as 
perceived  by  Anthony,  is  summarized  in  Table  5-1,  assuming  that 
each  type  is  a  point  on  a  continuum  ( Dumas ,1978) . 

This  framework  has  been  very  criticized,  however,  it  seems  that 
for  a  military  environment,  which  this  report  is  aiming  at,  it  still 
has  a  lot  of  relevance. 

Feltham  (1968)  defines  three  major  attributes  of  information, 
as  foil ows : 


Table  5-1.  Management  Information  Requirements  Accord  i  nv,  to  Anthony 


Appraisal  of  results  I  difficult  and  long  — - — —  »  less  difficult  and 

immediate 


Ke  levanre :  A  signal  is  relevant  if  its  receipt  changes  the 

dec  is  inn.  Therefore,  relevance  requires  specification  of 

« 

belli  a  decision  maker  and  a  decision.  An  ex-post  viewpoint 
oi  relevance  is  that  if  a  signal  changed  the  decision,  then 
the  information  provided  by  that  signal  was  relevant.  To 
he  an  effective  decision  criterion,  it  should  be  possible 
to  apply  the  concept  ex  ante. 

Time  1  iness :  Data  do  not  become  information  until  received 
by  the  decision  worker. 

IVu  additional  elements  are  associated  with  timeliness: 

* 

"reporting,  delay"  -  the  difference  between  the  time  of  the 
event  and  the  time  the  data  is  received,  and  "reporting 
interval"  -  referring  to  the  storage  of  data  and  reporting 
it  at  a  later  date. 

Accuracy- :  If  the  same  data  is  not  produced  every  time  the 

sane  event  occurs,  the  relationship  is  expressed  as  probability 
distribution.  These  differences  are  caused  by  errors  in 
recording,  processing  and  transmitting  the  data.  The  error 
bar.  two  basic  components:  bias  and  variability. 

the  above  definition  of  accuracy  is  basically  technical.  However, 
there  might  be  another  tvpe  of  error  -  the  "information 
perception  error"  -  in  the  case  where  the  same  data  does  not 
mean  the  same  thing  to  different  observers.  Thus,  it  is  not 
the  dat a  sent  to  the  decision  maker  which  is  important;  it  is 
the  decision  maker's  perception  of  the  meaning  of  the  data  which 
is  important. 
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Coin'll  (1471)  (Ictinus  five  attributes  (criteria)  of  an  MIS, 


.is  i  o  l  1 1 >ws : 

-  Relevance  -  the  first  and  paramount  attribute 

-  Timeliness 

-  Kconomv 

-  Accuracy 

-  Flexibility,  which  has  a  dual  purpose: 

•  flexibility  to  handle  growth 

•  flexibility  in  handling  inevitable  changes,  both  in 
planning  and  operation. 

It  should  not  be  surprising  that  there  is  some  overlap  in  the 
attributes  suggested  by  various  authors. 

Murdiek  and  Ross  (1975,  p.  357),  identify  the  following  attributes 
of  MIS: 

-  Purpose :  information  must  have  purpose  at  the  time  it  is 
transmitted  to  the  decision  maker 

-  .Hide  and  format:  mainly  documents,  verbal  material  or  visual 
(CRT) 

-  Redundancy :  the  excess  of  information  carried  per  unit  of 
data.  This  attribute  serves  as  a  safeguard  against  errors  in 
the  communication  process. 

-  Rate :  rate  of  transmission  may  be  represented  by  the  time 
required  to  understand  a  particular  situation. 

-  Frequency:  the  frequency  with  which  information  is  transmitted 
or  received  affects  its  value,  and  must  be  related  to  an 
operational  need. 
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T'l.iv  he  expressed  as  the  degree  of  confidence 


Re  I  i  ah  i  1  it-.-: 
t ho  decision  maker  places  in  the  information. 

-  Validity:  a  niea  nre  of  the  decree  to  which  the  information 
represents  what  it  purports  to  represent. 

Other  attributes  mentioned  bv  the  same  authors  include: 
accuracy  -  timeliness 

-elaritv  -  availability  on  demand 

-  distribution  -  selectivity  of  contents 

-  aj'prepr  i  nteness  -  disposition  method 

o:  detail  for  each 

-  cost  -  retention  time 

-  va Lue 

Tiie  Fort  Hood  Report  (1979)  lias  tiie  following  statement  concerning 

the  attributes  of  information: 

Informal  ion  was  considered  to  be  one  of  the  most  critical 
resources  in  the  management  process.  To  be  effective,  it 
must  be  timely,  accurate,  anti  supportive  of  the  decision 
making  process.  It  should  be  obtained,  stored,  analyzed  and 
used  in  as  economical  manner  as  possible. 

-l  is  statement  summarizes  a  military  user  observation  of 

in  format  ion  a  t  t  r i be  t  cs . 

Adding  a  lew  more  information  attributes  to  the  ones  mentioned 
above,  it  is  possible  now  to  recap  this  discussion  with  a  list  of 
"information  metrics."  in  doing  so,  it  was  found  useful  to  group 
these  metrics  in  tiie  three  major  elements  of  MIS,  namely: 

-  Management 
Information 

-  Systems 

s 
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Tab  1 1*  1-2  gives  this  list  of  information  metrics. 


Table  1-2 .  Management  Information  Metrics 


i  . . . 

|  Management 

Informat  ion 

Systems 

| 

!  -  Support  of  management 

-  Nature  of  Information 

-  Flexibility 

!  decisions 

-  Accuracy 

-  Adaptability 

!  -  I'ser  satisfaction 

! 

-  Redundancy 

-  Complexity 

-  Purpose 

| 

-  Reliability 

-  Structuredness 

-  Relevance 

-  Rate 

-  Distribution 

-  Timeliness 

-  Selectivity  of 

-  Frequency  of 

-  Validity 

contents 

transmission 

-  Frequency  of  use 

-  Clarity 

-  Tempos  of 

execution 

-  Availability  on 

-  Mode  and  Format 

demand 

-  Disposition 

method 

-  Time  horizon 

-  Retention 

-  bevel  of  aggregation 

method  time 

-  detail 

i  -  I  conomy/Cost 


I 
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MIS  SO.tWAKI.  1.1  Ki;0|- 1 VKNKSS  :  ASSESSMENT  OF  POSSIBLE  APPROACHES 


h  # 

(i .  1  Overv  iew 

As  inori'  and  more  organizational  resources,  especially  within 
the  military  system,  are  allocated  to  the  design  and  development  of 
MIS,  it  becomes  very  important  to  he  able  to  assess  the  effectiveness 
of  such  systems.  Existence  of  an  instrument  to  evaluate  such  systems 
would  he  verv  desirable,  as  each  user  could  denote,  using  the 
instruments,  how  he  viewed  his  MIS.  It  should  then  be  possible 
to  speoilv  to  tlit-  designers  and  maintainers  of  the  MIS  exactly 
where  and  how  enhancements  could  be  made.  Such  capability  would  be 
verv  desirable  t"  the  military  organization,  facing  a  dynamic 
informat  ion  environment  and  unstable  staffing  problem. 

ibis  .  hapter  investigates  and  assesses  the  currently  available 
tools  |,. i  evaluating  MIS  software  effectiveness.  In  doing  so,  one 
should  ki ep  l n  mind  the  various  comments  made  in  Chapter  A  about 
tin  decision  mukini'  process,  indicating  that  a  certain  technique 
will  ,  c. .  ii ! to  rent  e M e< t i veness ,  depending  on  the  organizational 
environment,  tnc  oi  ision  making  process  and  the  decision  maker. 

:  he  lewiiic  discussion  borrows  on  a  number  of  literature 
si'uri  >  ,  o-,;  c,  i.il'  -u  the  one  indicated  in  Chapter  3  as  major 

r  e  t  e  I  e  1 1 1  o'.. 

Bet  ore  |  r  ■  eeiiu,’  any  further,  it  is  worthwhile  to  note  some 
“t~ t •  1 1 1  s  n.idi  hv  Keen  and  Morton  (1P78)  on  the  evaluation  problems. 
Two  points  oi  view  can  be  identified  in  the  literature:  the  one 
incusing  on  Management  Information  Systems  (MIS),  and  the  one 
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focus  in);  on  information  Management  System  (IMS).  There  is  more 


than  a  semantic  difference  between  the  two:  IMS  implies  much 
more  concern  with  improving  the  operating  system,  usually  handled 
by  computer  scientists,  and  ignores  the  management  usefulness  of 
the  system,  a  problem  better  handled  by  management  scientists  and 
organizational  theorists.  Thus,  when  computer  scientists  talk 
about  MIS,  they  may  in  reality  have  IMS  in  their  mind.  This 
prompts  the  drive  to  increase  the  efficiency  of  systems,  which  may 
or  may  not  contribute  to  their  effectiveness. 

The  difficulties  associated  with  assessing  effectiveness  are 

summarized  by  Murdick  and  Ross  (1975,  p.  155). 

A  clearcut  method  for  measuring  the  costs  and  benefits  of  a 
new  MIS  has  not  yet  been  found. 

This  is  even  strengthened  by  Parden's  (1978)  comment  that 

...  tlie  cost  effectiveness  of  operations  systems  can  readily 
be  determined,  while  the  value  of  information  developed  in 
support  of  management  decisions  will  always  be  elusive. 

The  discussion  so  far  definitely  points  out  the  difficulties 
associated  with  effectiveness  evaluation,  however  it  also  amplifies 
the  importance  of  this  issue. 

As  a  pretext  to  MIS  software  effectiveness,  it  is  worthwhile 
to  gain  more  insight  into  MIS.  Dumas  (1978)  identifies  three 
"modes"  of  functioning  in  MIS,  as  follows: 

-  applications  oriented  mode 

-  data  base  mode 

-  informing  mode 
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Tli.-  three  modes,  among  other  features,  address 
respectivelv  the  structured,  semi-structured  and  less  structured 
decision  situations. 

In  the  application-oriented  mode,  MTS  provides  data  or 
measurements  within  a  crystal  1 ized  frame  of  reference  among 
organizational  partners.  The  data  base  mode  performs  integration 
of  data  in  broader,  less  crystallized  and  more  flexible  frame  of 
reference.  The  informing-mode  MTS  enables  a  goal  seeking  or  task 
oriented  decision  maker  to  select  or  change  frames  of  reference 
and  arrive  at  a  more  personal  appreciation  of  unstructured  problem. 
Mach  code  is  characterized  by  different  scopes  of  requirements  for 
data  and  information,  different  procedures,  and  different  types 
ot  evaluation.  Those  features  are  summarized  in  Table  6-1. 

It  should  be  noted  that  the  three  modes  are  not  mutually 
exclusive,  but  tend  to  be  on  a  Guttman  scale  (i.e.,  informing 
implies  data  base  implies  applications-oriented  mode). 

Another  aspect  of  MIS  evaluation  is  the  two  possible  "states 
of  beings"  of  the  system.  Davis  (1974)  recognized  the  evaluation  of 
a  new  or  replacement  svstem  versus  the  evaluation  of  exising  systems. 
Those  two  types  of  evaluation  correspond  respectively  to  the  stage 
of  design  and  operation  of  the  system.  Techniques  suitable  for  the 
first  stage  will  not  necessarily  be  suitable  for  the  second  stage, 
and  vice  versa.  Therefore,  this  distinction  has  to  be  made  when 
evaluating  different  approaches. 

The  various  available  techniques  for  evaluation  of  MIS 
effectiveness  can  be  classified  according  to  the  following  major 
groups : 


X 
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Fconom i c  I  v.i  ln.it  ion  Techniques 


-  Noneconomic  i'.v.i  i ua t  i on  Techniques 

-  behavioral 

-  mher 

-  M.in,i;..',eneii t  Science  Techniques 

i he  economic  techniques  include  all  those  where  a  dollar  value 
can  he  assigned.  Management  science  techniques  is  basically  the 
mathematical  modeling  approach  to  the  evaluation  problem.  All 
the  rest  n|  the  techniques  were  labelled  "non-economic "  where  some 
oi  them  are  "sein i -quant i l at i ve"  in  the  sense  that  numbers  can  be 
assigned,  and  the  rest  are  purely  qualitative. 

The  various  available  techniques  in  each  group  are  summarized 
in  Table  6-2,  which  considers  also  the  two  major  stages  identified 
in  MIS.  ibis  table  serves  as  a  guideline  for  the  discussion  in  this 
chapter.  First,  the  economic  techniques  are  discussed,  followed 
bv  the  non-cconomic  and  management  science  techniques. 

a  .  2  Kconomic  _Kval  nation  Techniques 

At  a  first  glance,  the  economic  evaluation  approach  seems  to 
be  verv  appealing,  -  for  a  couple  of  reasons.  First,  economic 
methods  are  popular,  well  understood  and  fit  with  other  organizational 
practices.  Second,  these  methods  refer  to  evaluations  expressed  in 
monetary  terms,  and  dollars  fulfill  the  ideal  objective  of  measurement 
on  a  ratio  scale  that  allows  to  compare,  order  and  compute  distances 
among  items. 

However,  one  should  realize  that  economic  reasoning  is  much 
concerned  with  the  optimal  allocation  of  scarce  resources,  and  therefore 


-t  .  t  t  M.  ■  .  ■  i  - .  •!>.  »!i  i .  <-v.tlu.tt  ion  will  he  directed  towards  efficiency 
tat  her  than  effectiveness,  i.e.,  towards  comparison  between 
t  < -venues  and  costs.  Further  more,  in  many  instances,  the  "economics 
* •  i  ini  one  it  ion  "  is  n< >_t  that  of  allocating  scarce  resources  i.e.  a 
tew  piece..  <•!  data  between  users  -  but  rather  that  of  eliminating 
o-.-i  rai-und.mt  in*  irrelevant  data.  Economists  have  Little  concern 
or  ispeot  of  in  i  onnat  ion  economics.  Cost-benefit  models  of 

MIS  evaluation  compute  costs  as  cost  of  information  services,  and 
the  bend  its  are  the  expected  desirability  of  the  outcome  -  a  very 
vauge  detinition.  Measuring,  the*  dollar  value  of  benefits  accrued 
I rom  informat ion  is  a  major  problem  as  many  benefits  are  "soft," 
and  not  eadilv  translatable  into  monetary  terms. 

According,  to  Keen  and  Morton  (  1978),  the  whole  area  of 
"i  nl  or:n.it  ion  economics”  is  a  small  and  ill  defined  one.  The  problem 
is  that  o!  measuring  information  value,  as  this  is  the  key  to 
consider i ng  alternatives  and  trade-offs.  At  the  present  there  is 
no  sat i  - i  k  Lory  method  to  evaluate  this  value. 

bearing  in  mind  the  above  comments  on  economic  evaluation 
techniques,  the  lol lowing  is  a  discussion  of  some  common  approaches, 
in  spite  oi  their  limited  usefulness.  The  ones  to  be  discussed  are 
cost  beneiit,  cost  saving  and  capital  budgeting. 

A  .  2  .  1  Cost  Benef i t .  This  method  is  a  mix  of  economic  evaluation 
of  costs  and  of  an  attempt  to  convert  to  hard  dollar  terms  benefits 
ot  the  svstem,  such  as  improved  decisions,  more  timely  information, 
etc.  Hi  is  approach  does  not  attempt  to  yield  maximizing  decisions. 


Hi  levant  costs  should  be  defined  as 


but  r  1 1  he  r  ::.l :  is  I  i  ■  i n j’  . 
oruudlv  .\s  possible,  and  all  benefits  should  be  included;  if 
it  is  imj.i's  .  i  s  |  e  to  estimate  them  directly,  a  surrogate 
i muster  price  mav  ho  generated,  or  a  judgment  can  be  used. 

during  the  I'hui's  i-osl -benef It  was  a  popular  approach, 
howev e r ,  in  :naitv  cases  it  proved  ineffectual  because  the  Intangible 
factors  can  rare!'-  he  converted  to  its  dollar  equivalent.  This 
is  even  more  difficult  in  a  military  environment.  Sassone 
and  Schafiei  (197b,  p.  44)  give  the  following  postulate:  "The 
value  of  a  project  Co  an  individual  is  equal  to  his  willingness 
to  pav  f  1 1  r  the  pro ject . " 

(n  .t  previous  work  by  Mason  and  Sassone  (1978),  some  of  the 
information  benefits  that  have  to  be  measured  were  defined  as 
f  o I  1 ows : 

-  liat a  quality 

-  but  a  reliability- 

intrinsic  value  of  information 

it  should  be  obvious  how  difficult  it  is  going  to  be  to  find 
oat,  in  a  military  system,  how  much  one  is  going  to  be  willing  to 
"pay"  for  the  above  benefits.  Even  using  a  surrogate  measure  such 
as  snudow  price  -  the  value  associated  with  a  unit  of  some  good 
indicating  how  much  some  unit  of  performance  can  be  increased  by 
the  use  of  marginal  use  of  that  commodity  -  is  not  going  to  be 
of  much  help  in  the  military  environment.  Even  the  cost  side  of 
the  analysis,  although  easier  to  measure  than  the  benefits,  still 
presents  some  assessment  problems. 
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Final lv,  it  shonlil  be  recognized  that  the  methodology  for  cost- 
benefit  analysis  should  he  different  during  the  design  phase  and  the 
operation  phase.  Hie  problem  during  the  design  phase  is  that  of 
defining  alternatives  and  assessing  the  costs  and  benefits.  During 
t he  operation  phase  the  problem  is  that  of  allocating  costs  and 
measuring  benefits  of  the  selected  alternative.  Thus,  cost-benefit 
analysis  would  be  a  much  stronger  tool  for  the  design  phase. 

In  summary,  the  following  assessment  can  be  made  of  this 
approach : 

-  Cost-Benefit  Analysis  (CBA1  is  a  well  established  technique 

as  such 

-  Enables  explicit  definition  and  examination  of  alternatives 

-  Measurement  of  benefits  is  difficult,  especially  within  a 

mi  1 itarv  MIS 

-  Effectiveness  is  measured  indrectlv  with  dollar  value 

-  1  in*  methodology  for  applying  CBA  in  a  military  MIS  has  to 
be  developed 

-  It  scorns  that  the  approach  is  expensive  and  time  consuming. 

b.2.2  Cost  Savings  Approaches.  Techniques  in  this  group  include 

two  major  categories: 

-  savings  due  to  better  decisions,  and  attributed  -  supposedly  - 
to  improved  performances  of  the  information  system. 

-  savings  internal  to  the  information  system  which  do-not  modify 
the  decision. 

Savings  of  the  second  type  (such  as  reduction  of  clerical  effort) 
are  basically  ef f 1c iency  measures,  although  It  is  not  uncommon  to  see 
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i  huso  approaches  used  as  effectiveness  measures  for  MIS. 

:  -av  i  nt'.s  of  i  he  first  tvpo  are  theoretically  very  appealing, 
iiovovs  r.  it  is  practically  almost  impossible  to  measure  them. 

Furl  iu  nm'ir,  those  cost:  savings  do  not  give  any  control  of  the 
virion-  at  tributes  ol  information,  such  as  timeliness,  relevance, 
linn.io.  >io. 

It  seems  that  the  highest  appeal  of  those  methods  is  their 
s imp  1 i o 1 1 v .  On  the  other  iiand,  their  usefulness  in  evaluating 
el  feet i voness  is  very  limited. 

In  suinmarv,  the  following  assessment  can  be  presented: 

-  'Hie  major  advantage  oi  the  cost-savings  approach  are  its 
coni  op tun  1  simplicity 

Its  use  usually  emphasizes  input  economization  rather  than 

value  ol  MIS 

-  Cost  savings  directly  attributable  to  MIS  are  hard  to 
e  s  t i ma t c 

-  For  use  in  military  systems,  non  dollar  savings  are  oftentime 
more  important. 

-  Many  important  information  attributes  are  not  controlled. 

P.J.  1.  Capital  Budgeting  Methods.  Those  methods  could  be 

regarded  as  an  extension  of  the  cost  savings  approach  discussed 
previously.  In  a  way,  the  capital  budgeting  methods  are  the  tools 
to  perform  the  proper  economic  analysis  on  the  cash  flows  generated 
hy  the  savings  methods.  tn  this  category  one  could  include 
discounted  cash  flow  method,  pay  back  period,  break  even  analysis, 
internal  rate  of  return  etc.  The  techniques  per  se  are  well  developed 
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uiil  r.-cogn  i  /.uil ,  however,  Lheir  use  is  dependent  on  being  able  to 


generate  i  lie  proper  cash  flows,  anil  that's  where  the  major  problem 

i  s  . 

o.i.  Non-  1-Vonomii-  Kvaluat  ion  Techniques:  Behavioral 

Kiveii  the  various  shortcomings  of  the  economic  evaluation 
methods  in  the  context  ol  MIS,  it  is  only  natural  that  attention 
n.is  been  .’ocu.-eil  on  non-eoonomic  approaches.  This  section  will 
discus  .  1 1  >  havioral  techniques,  whereas  the  next  one  handles  other 

f  «  t'itM  ;  siir:i  . 

None,  ouomie  methods  are  in  general  more  empirical.  Their 
;/r«.  sent  state  <>:  tiie  art  is  less  developed  than  that  of  the  economic 
approach. s .  (he  rationale  for  using  non-economic  methods  is  that 
they  can  allow  accounting  for  other  benefits  that  can  not  be 
considered  in  the  economic  evaluation  approaches.  Furthermore, 
i  h.se  methods  are  iii  general  be  L  ter  in  evaluating  the  operation  stage 
•i  "a:  .  Tncir  main  advantage  is  that,  if  a  user's  point  of  view  is 
to  :.<•  ijrcii  in  evaluating  MIS  effectiveness  (as  is  done  in  this 
research;  then  1  lie  noneconomic  methods,  especially  the  behavioral, 
are  user's  oriented  in  nature.  It  has  been  shown  that  paying  attention 
onlv  in  "computer"  issues  is  not  nearly  enough  for  success  of 
ini ormat ion  systems  in  an  organization  (Keen  and  Morton,  1978,  p.  50). 

1  he  major  criticism  Leveled  at  the  behavioral  methods  is  their 
lack  ol  objectivity.  However,  objectivity  is  not  always  a  valid 
con. apt  when  dealing  with  socio-teelmical  systems  such  as  MIS. 

In  the  balance  of  this  section,  six  different  behavioral 
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I  i  i  im  i  <ju«  •  ■ .  .in-  discussed,  .is  follows:  user's  satisfaction,  manager's 
assessment  of  the  system's  value,  the  decision  maker  learning 
process,  decision  process  changes,  expert  opinion,  anecdotal 
evidence . 

b.i.i.  User's  Satisfaction.  This  is  possibly  the  most  important 
single  parameter  of  MTS  that  is  to  be  measured.  The  importance  of 
this  parameter  stems  from  the  fact  that  it  represents  some  integration 
of  ulj  the  information  attributes,  as  perceived  by  the  user .  User's 
satisfaction  can  be  measured  through  some  form  of  psycho-social 
inquire,  or  attitude  rating. 

One  of  the  problems  of  evaluating  user's  attitude  is  the 
variability  of  this  attitude  from  user  to  user,  and  for  the  same 
user  for  different  circumstances  and  time. 

Some  recent  research  in  this  area  will  first  be  cited.  Walther 
(1971)  recognized  the  problem  mentioned  above  and  defined  the  concept 
of  "flexibility,"  which  is  the  "capability  of  the  system  being  changed 
by  the  user  in  order  to  make  it  responsive  and  adaptable  to  ever- 
changing  user  needs  and  pref erences . "  He  also  recognized  that  flexibility 
is  not  uniformly  good  tor  everyone.  The  major  contribution  of  this 
research  is  showing  that  semantic-differential  scales  are  feasible  for 
evaluation  of  user  attitudes. 

Semantic-differential  has  been  used  also  by  Gallagher  (1974),  who 
found  it  a  useful  tool  for  measuring  and  analyzing  the  qualitative 
value  of  MIS.  Some  of  his  other  findings  are  of  interest  too: 

-  A  reasonable  estimate  of  the  monetary  value  of  a  specific 
management  information  can  be  determined  by  asking  each  user 
to  estimate  the  system's  value  to  himself. 
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I mage  stair  is  not  reliable  measure  of  the  nonmonetary 
v  !  1 ue  o  t  MIS. 

"•■•ward  (147  1)  devised  and  tested  a  questionnaire  using  Likert 
seal  os,  and  .•(included  that  "measuring  user  satisfation  with  the 
gropes *.,j  information  system  is  a  feasible  substitute  for  measuring 
ini  I'lT.i.it  n'ii  system  effectiveness." 

i.e  above  examples  are  representative  of  current  approaches, 
as  1 ■  -po i 1 1 'd  in  the  literature,  to  measuring  satisfaction.  As  much 
as  it  looks  promising,  there  are  certain  drawbacks.  All  those 
approach.-  >  are  relatively  recent,  and  have  not  yielded  much 
enthusiasm  from  practitioners.  All  use  the  techniques  of  questionnaires 
which  time  and  again  are  criticized.  To  overcome  this  difficulty, 
unobtrusive  techniques  have  been  tried,  such  as  automatic  recording 
ot  tho  riiuracterist ies  of  users  (Kitous,  1976).  Again,  those  methods 
wer<-  i.-vei  tried  in  a  large  scale  svstem. 

!o  recap,  there  are  not  many  instances  of  systematic  use  of 
oho-  ;oc i a l  techniques  for  evaluating  computer  application  performance. 
Vet  ,  1 1 ■  r  ’  s  opinion:;  are  a  valid  surrogate  for  measuring  MIS 
.•I t  ert  i vencss.  However,  the  methodology  for  pshyco-social  inquiry 
is  tar  i  rom  being,  established,  to  the  point  that  it  is  hard  to  say 
whether  scientific  knowledge  is  attainable  in  this  field.  Even  if 
an  agreed  upon  method  were  available,  there  are  still  practical 
d  i  t  t  i  v  u  i  t  i  es  of  implementation,  such  as  iiigh  cost  and  lack  of 
skilled  personnel. 

In  summary,  the  assessment  of  tiiis  approach  is: 

-  Hie  general  concept  is  very  appealing,  as  user's  satisfaction 

S 
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could  g  ivi>  one  overall  measure  of  MIS  software  effectiveness. 


The  basic  methodology  for  this  approach  has  mot  matured  yet. 

It  seems  that  no  large  scale  MIS  has  been  studied  using  this 
methodol ogv. 

-  Costly  and  time  consuming  (for  interviewers  and  interviewees) 

-  Risk  of  bias  due  to  the  behavioral  impact  of  the  approach, 
besides  the  familiar  bias  due  to  sample  size  and  representativeness. 

-  Difficult  to  conduct  on  a  periodic  basis. 

n.3.2  Manager's  Assessment  of  the  System’s  Value.  Asking  manager's 
is  on.’  effective  wav  of  defining  the  system's  value.  Their  perceptions 
can  be  gathered  at  regular  intervals,  using  questionnaires  or 
structured  interviews.  One  such  attempt  is  reported  by  Swanson  (1974) . 
lie  used  questionnaires,  and  measured  user's  appreciation  by  averaging 
user's  ovluations  of  the  information  they  received  and  the  means  by 
which  the  information  was  provided.  The  result  was  an  index  of 
appreciation.  The  study  was  performed  in  a  real  world  situation, 
however,  it  was  directed  at  a  specific  information  system  and  therefore 
could  not  be  generalized. 

This  approach  can  be  viewed  as  a  subset  of  the  previous  one,  thus 
the  -same  comments  apply  here  too. 

6.3.1  The  Decision  Maker  Learning  Process.  This  approach  utilizes 
research  done  in  cognitive  processes  and  applies  it  to  MIS.  It  is  an 
attempt  to  evaluate  the  human  information  processing  and  learning  and 
use  it  as  an  indirect  measure  of  MIS  effectiveness.  This  method  requires 
the  use  of  simple  diagnostic  techniques  for  capturing  the  decision 
maker's  concepts  and  learning.  However,  at  present,  the  use  of  such 
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techniques  in  tin-  real  world  is  close  to  zero  (Keen  and  Morton,  1978, 
p.  .M'O.  Measuring  this  cognitive  process  can  be  attempted  by  using 
quest  i  own  i  res,  requiring  that  a  "before"  and  "after"  condition  be 
•  ab  1  i  shed .  liven  if  the  process  of  learning  can  be  measured,  it  is 
’ ia rd  to  place  1  value  on,  as  the  training  and  tools  for  it  are 
del  ic  ient  . 

It  is  obvious  that  Lliis  tool  could  be  adequate  for  the  operation 
phase  of  MIS.  However,  a  lot  of  behavioral  research  has  to  be 
;uTi  orno.  in  developing  the  tool  per  se. 

decision  r recess  Changes.  In  this  approach,  the  outcome 
o:  Lie  eei:  ion  is  .econdarv,  and  the  element  of  interest  is  the 
■hens  .  .n  the  decision  process,  where  the  implication  is  that  better 
dec  i-  ion:  will  lie  made  if  the  decision  making  process  is  improved. 

A  si  in,  this  approach  too  is  suitable  for  the  operation  phase  of  MIS. 

This  mo i hod  requires  two  problems  to  be  resolved  before  it  can 
e  implemented: 

-  del  iniLion  of  a  "better"  decision  process 

-  i-.casiir  i  ng  the  process  changes 

Overcoming  the  first  problem  requires  a  normative  model  that 
dot  Lucs  u  "better"  decision  process.  The  measurement  problem  is  a 
more  "knotty"  one.  An  attempt  to  tackle  this  problem  is  reported 
by  Stain  !  L  (1974)  and  flinzberg  (1975),  who  used  "traces"  to  measure 
changes  in  the  decision  process.  A  trace  is  a  record  of  interaction 
between  the  decision  maker  and  the  MIS.  Those  traces  are  easy  to 
record  when  the  interaction  is  done  through  a  terminal.  However, 
one  should  not  try  to  implement  them  without  informing  the  users,  as 
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thov  represent  some  sort  of  surveillance. 

Traces  are  one  of  the  more  powerful  methodologies  developed  so 
tar  that  provide  insight  to  the  qualitative  aspects  of  the  decision 
making  process.  One  should  realize,  however,  that  traces  are  not  a 
mere  count,  and  requires  further  analysis. 

6.1.5  Expert  Opinion.  This  is  a  typical  approach— use  of  a  1 

reference  group-to  evaluate  situations  where  standards  of  desirability 

are  ambiguous.  The  leading  technique  in  this  category  is  the  Delphi 
method.  I.igon  (1978)  reports  the  use  of  this  method  in  the  context 

of  MIS,  in  order  to  identify  the  ingredients  of  a  successful  system.  j 

i 

The  Delphi  methodology  as  such  is  well  developed,  however,  its  , 

application  in  MIS  is  relatively  new,  and  requires  more  development. 

Furthermore,  utilization  of  Delphi  within  a  military  system  should  be 

i 

approached  verv  cautiously. 

,  I 

6.1.6  Anecdotal  Evidence.  This  is  a  method  that  is  intended  to  j 

supplement  formal  evaluation  by  collecting  anecdotal  evidence  such  as  ! 

j  , 

insights,  examples,  lessons  learned,  opinions  and  events  collected  by  j 

i  l  \ 

,  a  trusted,  neutral,  skilled  observer.  Quantifying  the  results  is 

,  next  to  impossible  in  this  approach,  and  the  outcome  depends  heavily 

!  s  ! 

,  |  on  the  observer. 

1  !  * 

6 .4  Noneconomic  Evaluation  Techniques:  Other 
|  This  group  includes  a  number  of  techniques,  most  of  them  might 

be  labeled  "quantitative"  in  the  sense  that  they  contain  numbers, 

!  however,  they  are  noneconomic  in  nature.  Those  techniques  that  ! 

contain  numbers  mainly  use  the  basic  approach  of  mere  count  of 

physical  items,  and  can  be  summarized  in  three  types:  volume,  time 

i 

I  and  checklist.  Additional  methods  discussed  in  this  section  include 
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M  iviiT  measure  .mil  appraisal  bv  comparison. 


>1.4.1  Time  Me  (hods .  Time  is  one  of  the  few  elements  that  are 
c.isv  t  <1  measure  and  understand ,  and  it  may  be  an  attribute  of  the 
inform_.it  ion  system  or  in  formation  itself.  Also,  time  may  represent 
a  measure  of  efficiently  or  effect iveness .  Thus,  evaluating  MIS 
software  bv  the  number  of  reports  processed  per  unit  of  time  is 
definittlv  a  measure  of  ef f iciency ,  not  effectiveness.  However, 
timeliness  is  one  of  the  important  attributes  of  information  that 
contributes  to  MIS  effectiveness. 

i  ime  1  i ness  can  he  measured  in  terms  of  the  difference  between 
the  tine  requires!  by  the  decision  maker  and  the  actual  time  when  the 
information  is  provided  by  the  MIS.  This  difference  may  be  negative 
(information  delay)  or  positive.  This  delay  can  be  a  design  parameter 
for  MIS,  or  a  monitoring  element  during  operation. 

Although  easy  to  measure,  it  is  difficult  to  state  cause-effect 
relationship  between  more  timely  information  and  better  decisions. 

h._4.2  Volume  Methods.  Most  of  the  volume  measurements  -  such 
as  inputs  and  outputs  -  definitely  relate  to  measuring  efficiency 
rather  than  effectiveness,  in  many  cases  that  of  the  hardware  system. 
However,  there  are  instances  where  volume  is  used  for  evaluating  the 
effectiveness  of  MIS.  This  is  considered  in  a  research  reported  by 
f.'oidb ere.  (1971),  Kennedy  and  Maliaparta  (1975)  and  Kitous  (1976),  where 
the  effectiveness  of  MIS  was  evaluated  bv  counting  how  many  times  a 
given  report,  or  a  piece  of  information  is  used  by  the  decision  maker, 
or  how  often  items  in  the  data  base  are  accessed,  or  how  important  is 
the  ratio  of  useful  information  to  noise. 
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Since  volumes  may  measure  the  functioning  of  the  system  under 
different  conditions,  they  should  be  presented  as  statistical  data 
and  not  as  absolute  data. 

The  volume  approach  is  definitely  an  MIS  operation  phase  approach. 

It  has  the  advantage  of  being  easy  to  measure  and  understand.  On  the 
other  hand,  the  correlation  between,  say  high  volume  of  report  use 
and  high  effectiveness  of  information  does  not  always  hold  true,  as 
the  high  usage  might  be  simply  due  to  the  lack  of  any  other  tool,  and 
not  necessarily  due  to  the  MIS  effectiveness. 

6.4.3  Checklist  Methods.  All  in  all,  a  relatively  weak  approach, 
that  fits  most  structured  decision  environment.  The  method  is  composed 
of  a  list  of  criteria  and  a  measure  of  its  achievement,  such  as  cost 
control  ach ievements ,  number  of  personnel  trained,  number  of  reruns, 
etc.  These  criteria  can  be  placed  in  a  historical  perspective  showing 

t  rends . 

6.4.4  Service  Measure.  This  approach  is  suggested  by  Keen  and 
Morton  (1978),  and  is  composed  of  the  following  elements: 

-  responsiveness  of  the  system 

-  availability  and  convenience  of  access 

-  reliability 

-  quality  of  system  support,  such  as  documentation  and  training 

Those  attributes  could  be  very  helpful  in  monitoring  MIS.  However, 

the  authors  give  no  clue  as  to  how  to  perform  the  measurement  of  those 
elements . 

b.4.5  Appraisal  by  Comparison.  This  approach  is  111  defined,  very 
little  recognized  in  the  literature,  and  is  composed  of  comparing  the 
performance  observed  with  the  performance  in  similar  organization.  There 


i  no  r  i  ;■  .1'  n  .'r;  i:h-i  IhhIh  1  belt  ir.d  tills  approach,  especially  for  MIS, 
and  it  n.  i ;1 1 1  he  looked  upon  more  as  a  concept  rather  than  as  a  method. 

<>.  ’>  Man;:o  -went  _So  ience  [’.valuation  Techniques 

ri:is  roup  of  methods  is  mainlv  beneficial  during  the  design  phase 
of  MI !' .  Management  Science  techniques  imply  some  sort  of  mathematical 
model  building  manipulated  through  optimization,  simulation  or  heuristics. 
Topical  iv,  the  model  includes  some  of  the  MTS  objectives  as  related  to 
the  total  organization.  The  major  shortcomings  of  this  approach  are: 
orientation  towards  economic  data  analysis,  with  all  the 
limitations  encountered  in  MIS 

-  hard  to  understand  by  managers 

-  based  upon  assumptions,  which  are  often  erroneous,  due  to  the 
difficulty  of  the  socio  environment  of  MIS. 

-  validation  of  results  is  often  time  hypothetical,  as  models  are 
seldom  tested  in  a  real  setting. 

in  spite  of  the  above  difficulties  management  science  approach 
may  have  some  merits  in  certain  cases.  Three  approaches  will  be 
.urveved :  simtiiat.  ton,  model  building  and  risk  analysis  and  sensitivity 

anal  vs  is. 

().■).  I  Simulation.  Two  major  tvpes  of  simulation  models  exist.* 
for  hardware  evaluation,  and  for  ccxnputer  systems  application  evaluation. 
In  the  context  of  the  present  discussion,  only  the  second  type  is  of 
interest,  as  it  engulfs  the  evaluations  of  the  computer  and  of  its 
decision  making  environment.  This  approach  was  first  launched  by 
bonini  (1961)  wbo  proposed  to  relate  organizational  behavior  and 
informational  factors  to  the  economic  variables  in  the  firm.  Thus, 
some  simulation  models  possess  many  of  the  shortcomings  of  economic 
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evaluations . 


Boyd  and  Krasnow  (1963),  constructed  a  simulation  model  which 
basically  evaluated  the  timeliness  of  information.  Kriebel  (1969) 
used  simulation  models  to  evaluate  the  joint  MIS  -  decision  system. 
Courbun  (1976)  followed  this  approach  in  the  area  of  production 
information  systems  and  developed  a  simulator  (MISSIM). 

Other  applications  of  simulation  can  be  found  in  the  literature, 
and  the  field  of  application  is  varied.  As  much  as  it  looks  appealing, 
MIS  simulation  seems  not  to  have  left  the  research  environment  on 
into  the  real  world.  Most  reports  come  from  researchers,  especially 
doctoral  students.  Probably  one  of  the  major  problems  in  real  world 
applications  is  that  the  input  data  is  not  readily  available,  costly 
and  unreliable.  Furthermore,  simulation  requires  some  measurement 
on  a  real  system  with  statistical  significance  -  a  costly  proposition 
somet imes . 

In  summary,  MIS  simulation  is  a  sound  concept  that  requires  further 
development  if  an  attempt  is  to  be  made  to  use  this  approach  in  the 
context  of  military  MIS. 

b.5.2  Model  Building.  For  the  completeness  of  the  discussion  here, 
it  is  worthwhile  to  review  an  example  of  structuring  a  mathematical 
model  of  MIS.  The  example  presented  here  is  taken  from  Kennedy  and 
Mahapatra  (1975). 

Notations: 

Suppose  there  are  "m"  number  of  factors  given  by  f^,...,fm(or  pieces 
of  information  or  data)  that  are  affecting  all  the  decisions  of  the 
organization. 

Suppose  there  are  "n"  number  of  decisions  given  by  T>^,...,Du  made 

/ 
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in  i-.ii  h  department  or  organizational  subunit. 

Suppose  there  aro  "r"  number  of  organizational  subunits  or 

tU  partments  in  the  system  given  by 

Then  in  general  f...  denotes,  information  element  "i"  affects 
tjk 

(tir  is  needed  for  )  decision  "j"  in  department  "k"  where 

x  1 , .  .  .  ,m 
j  =  1 ,  .  .  .  ,  n 
k  =  l,...,r 

I.et  "  represent  the  frequency  of  decision  "j"  in  department  "k." 

J  k 

Let  "i  "  be  the  importance  (rank)  of  factor  "i"  in  decision  "j"  in 
department  "k." 

Let  "b be  the  importance  (rank)  of  decision  "j"  in  department  "k." 

J  k 

I.et  "y, "  be  the  importance  (rank)  of  department  "k"  in  the  system. 

k 

Then  the  importance  (rank)  of  factor  Mf .  M  for  decision  "j"  in  department 

1 J  k 


a  ...  (3a)  ..  Y,  =  , 

i.jkv  jk  k  ljk 


t . . .  =  x . . , / y  a. .. 

ljk  ijk  £  ljk 


■  “jk’jk^  ^k'bi 


Therefore,  the  importance  (rank)  of  the  information  element  "f/1  in  the 


total  system 


\=i  j=i  ('*1-ik  ‘  (S"a)Jk  ’  Yk) 


=  l  £  “ilk 

k=l  |=1  1,k 
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-  ii  .  (sav)  i  =  l  , .  .  .  ,m 

From  the  preceding  equations,  it  is  apparent  that  the  total 

importance  of  all  information  elements  summed,  must  equal  unity  times 
the  number  of  departments.  From  this  it  follows  that  each  p ^  is  a 
measures  of  relative  importance.  Also,  the  sum  of  the  values  of  those 
information  elements  now  provided  can  easily  be  interpreted  in  terms 
of  efficiency  in  meeting  total  (ideal)  information  needs. 

Thus,  the  process  allows  us  to  arrive  at  the  importance  index 
(p^)  of  all  (information)  factors  that  need  be  kept  in  the  MIS. 
Depending  on  the  budgetary  constraints  and  the  computed  ranks  of  all 
the  information  elements,  the  inclusion  or  exclusion  from  the  data 
base  of  MIS  may  be  determined. 

At  best,  it  can  be  said  that  if  an  optimization  model  can  be 
formulated  it  will  evaluate  a  very  narrow  segment  of  MIS. 

b.3.3  Risk  and  Sensitivity  Analysis.  Should  be  looked  upon  as 
a  supplementing  approach  to  simulation  or  optimization. 

Sensitiv  l/  is  an  attempt  to  identify  critical  variables 
recognizing  the  fuzziness  of  MIS  environment,  or  low  reliability  of 
data,  or  the  simplifying  hypotheses  underlying  the  model. 

Risk  tries  to  quantify  identified  weaknesses  of  the  system, 
such  as  risks  of  delays,  errors  and  underestimation  of  costs,  or 
uncertainty  of  environment. 

This  is  more  of  a  general  concept.  Spec  i  f  ic  application  have  to 
be  "tailor  made"  according  to  the  type  of  MIS  and  environment. 

A  summary  of  the  various  techniques  and  their  relation  to 
information  metrics  is  discussed  in  the  following  chapter: 
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'1AN.V ;!  "I  V!  I N l-'ORMAT  I DN  METRICS  AND  EV AM' ATI  ON  TECHNIQUES 


Tin.*  discussion  so  t  .u  nmicntr.itwl  on  defining  and  assessing 
tin  various  available  evaluation  techniques  for  MIS  software 
of t i ,  t i veiiess.  However,  one  dimension  is  still  missing,  namely, 
the  extent  to  whieh  available  techniques  can  evaluate  management 
ini  orm.it  ion  attributes.  This  chapter  performs  such  analysis. 

In  order  to  proreed,  it  is  worthwhile  to  do  some  additional 
grouping  "i  the  attributes  shown  in  Table  5-3.  This  grouping  is 
shown  in  Table  /-I. 


T-ible  /-I.  Management  Information  Metrics  -  Regrouping 


Mann  gotiien  t 

Information 

r- — - - 

Systems 

i  :  1  >e,  i  s  i  on  Mak  i  ng 

I:  Content 

1:  Structure 

-  support  of  mgt  . 

'll  O  1 :  ions 

-  user  satisfaction 

•  p'. I  Pose 

-  relevance 

-  vn  i  i  d  i  t  v 

-  level  oi  agglegation 
ie  t  a  i  1 

-  nature  of  information 

-  accuracy 

-  redundancy 

-  reliability 

1 

i 

-  flexibility 

-  adaptability 

-  complexity 

-  structuredness 

11:  T  L;:ie 

1 

1 1 :  Presentat ion 

II:  Speed 

-  timeliness 

-  frequency  ol  use 

-  availability  on  demand 

-  time  horizon 

i 

-  selectivity  of 
contents 

-  clarity 

-  mode  and  format 

-  frequency  of 
transmission 

-  tempos  of 
execution 

III:  Economy 

I  r  I  :  Rate 

III:  Distribution 

-  fconmiiy/rost 

-  rate 

-  disposition  method 

-  retention  time 

/ 
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Tims,  t ho  grouping  of  the  management  information  metics  can 


ho  .summarized  as  follows: 

Table  7-2.  Grouping  Summary 


Group 

Management 

Information 

Systems 

i 

Decision  Making 

Content 

Structure 

1 1 

Time 

Presentation 

Speed 

III 

- -  .  ._ 

Economy 

Rate 

Distribution 

Some  general  observations  concerning  the  above  grouping  are  in 
order.  One  interesting  approach  is  to  examine  the  above  attributes 
i rom  the  point  of  view  of  the  type  of  evaluation  technique  required, 
namely:  subjective  -  where  a  high  element  of  judgement  (i.e., 

behavioral)  is  required,  or  objective  -  where  more  direct  measurement 
is  applicable.  This  is  done  in  Table  7-3  where  another  group  was 
added  -  mixed  evaluation,  when  direct  measurement  and  judgement  are 
requ i red . 


Table  7-3.  Type  of  Evaluation  Required 

(S  -  subjective,  0-  Objective, 
M  -  Mixed) 


Group 

Management 

Information 

Systems 

1 

S 

0 

M 

LI 

0 

M 

0 

I  IT 

M 

0 

0 

The  point  of  departure  for  this  research  was  taking  a  user’s 
point  of  view  to  evaluate  MIS  software  effectiveness.  This  immediately 
creates  the  image  of  a  more  subjective  evaluation  approach.  What 
emerges  from  the  above  analysis  is  that  only  one  group  of  management 
information  attributes  -  as  important  as  it  is-requires  a  pure 
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In'll. iv  i  ora  I  approach,  wlicrcas  the  rest  of  them  need  a  mixed  or 
[Hire  objective  approach.  The  importance  of  this  observation  is 
that  the  behavioral  approaches  are  deficient  because  of  immature 
methodologies  (sometimes  labeled  "soft"  or  ’’weak”  techniques), 
where  for  objective  evaluat ions  more  rigorous  techniques  can  be 
used  or  developed. 

Further  insight  can  now  be  obtained  by  examining  the  management 
ini  urination  metrics  coverage  achieved  by  the  various  techniques 
presented  in  this  chapter.  This  is  done  in  Table  7-4,  which  also 
summur i  lies  the  discussion  in  this  chapter.  For  indicating  the 
level  of  coverage  of  a  group  of  attributes  by  a  technique,  a  scale 
of  three  numbers  was  used,  representing  the  following: 

1  -  Primary  applicability  of  the  technique  to  measuring  the 
a 1 1  r i bate 

-  Secondary  relation  to  the  attribute 

i  -  Weak  relation  to  the  attribute. 

Some  comments  about  Table  7-4  will  enhance  its  understanding, 
ilii  entry  to  the  table  is  through  the  "Evaluation  Technique"  column. 
The  right  hand  side  of  the  table  ranks  each  technique  in  terms  of 
its  appl icabi 1 i t v  to  measuring  the  specific  metric  group.  The  left 
hand  side  indicates  the  MIS  phase  in  which  the  technique  is 
applicable,  and  also  assesses  the  major  advantages  and  disadvantages 
'>1  each  evaluation  technique.  In  reviewing  this  table,  one  should 
inspect  both  sides.  Thus,  for  example,  user's  satisfaction  is 
identified  as  verv  applicable  to  measuring  decision  making  attributes 


liable  7-1).  However,  her, au.se  of  the  weakness  of  the  basic 
evaluation  methodology,  the  net  result  is  inadequate  evaluation 
of  the  attribute. 

The  emerging  picture  from  Table  7-4  is  that  the  field  of 
evaluating  MIS  software  effectiveness  -  a  user's  point  of  view 
although  having  many  shortcomings,  seems  to  be  in  a  better  shape 
than  represented  often  times  in  the  literature  by  comments  such 
as:  "A  clear  cut  method  for  measuring  the  benefits  of  MIS  has 

not  yet  been  found"  (Murdick  and  Ross,  1975). 

Spec  if ica  1  1 v ,  it  seems  that  the  "decision  making"  group  of 
attributes  received  more  attention  than  the  "information"  and 
"system"  groups,  although  without  very  much  success.  On  the 
other  hand,  the  "information"  and  "system"  groups  have  a  potential 
lor  better  results  because  of  the  nature  of  the  measurement 
techniques  required  -  less  reliance  on  behavioral  science  techniques 
and  more  reliance  on  objective  approaches  (Table  7-3). 
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8.  MIS  SOFTWARE  EFFECTIVENESS:  WHERE  DO  WE  GO  FROM  HERE? 


8 .  I  I  ni  roilin'  t  ion 

In  order  to  he  able  to  assess  research  needs  in  MIS  software 
effectiveness,  it  is  not  sufficient  to  establish  the  "state  of  the 
art"  of  the  current  approaches,  but  other  elements  affecting 
those  needs  have  to  he  examined.  Specifically,  the  issues  of 
requirement  planning,  future  trends,  centralization  versus 
decentralization  of  computer  facilities  and  the  MIS  design  process 
are  going  to  he  discussed. 

8  A  _  Requi remen t  Planning 

The  purpose  of  the  discussion  here  is  not  to  make  a  thorough 
analysis  of  this  process,  but  rather  describe  a  few  aspects  of 
requirement  planning  and  point  their  importance  to  the  MIS 
evaluation  process. 

Requirement  planning  or  need  identification  is  a  crucial 
element  in  MIS  design,  however,  it  is  also  as  important  for  the 
evaluation  process,  since  a  clear  statement  of  needs  can  facilitate 
measurement  of  their  fulfilment  ex  ante.  It  is  one  of  the  most 
important,  vet  one  of  the  most  difficult  areas  of  MIS,  since 
requirements,  like  beauty,  are  often  times  in  the  eye  of  the 
beholder.  Even  more  so  when  a  few  decision  makers,  performing 
the  same  managerial  function,  are  asked  to  identify  their  needs. 

The  type  of  needs  a  decision  maker  has  at  various  times  and 
for  various  purposes  depends  largely  upon  the  personal  attributes 
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iikI  the  organ i  zat  ional  environment  in  which  decisions  are  made 
(Murdick  and  Ross,  1973). 

Personal  attributes  influence  the  needs  definition  through 

three  elements: 

-  Knowledge  of  information  systems  -  The  more  the  decision 
maker  knows  about  computer  based  systems,  the  more 
sophisticated  and  specific  his  needs  are  going  to  be. 

-  Decision  making  style  -  affects  the  kind  and  amount  of 
information  required.  Here  comes  to  bear  the  various 
"inquiring  systems"  as  defined  in  Chapter  4. 

-  Perception  of  information  needs  -  one  common  problem  is 
that  many  decision  makers  are  ignorant  of  the  type  of 
information  they  need. 

Organizational  environment  interacts  with  the  needs  definition 

as  toll ows : 

-  Nature  of  the  organization  -  the  larger,  more  complex 
organizations  require  more  formal  information  systems,  which 
are  critical  to  their  operations. 

-  bevel  of  management  -  in  Chapter  5  the  various  information 
needs  of  the  three  m  inagement  levels  (Anthony,  1965)  were 
defined.  Each  level  needs  different  types  of  information, 
iu  different  form,  different  amount  of  detail  and  different 
frequency.  Furthermore,  decision  makers  at  all  levels  have 
different  information  needs. 

~  Structure  of  the  organization  -  the  more  highly  structured 
the  organization,  the  easier  it  is  to  define  information 
needs.  This,  in  a  way,  should  make  the  requirement  planning 
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process  easier  within  a  military  organization. 

A  systematic  approach  to  requirement  planning  is  important 
in  any  organization,  a  military  organization  thus  included.  One 
excellent  example  of  identifying  needs  within  the  Army  system  - 
at  tlie  installation  management  level  -  is  given  in  the  Fort  Hood 
IMS  report  (March,  1979).  The  method  used  there,  to  identify 
needs  and  other  parameters  of  the  system,  is  a  "Four  Quadrant 
Matrix,"  schematically  presented  in  Figure  8-1. 


functions  and  subfunctions 


DATA  SYSTEMS 


QUADRANT  IV 

How  is  efficiency  of 
function  performance 
measured 
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QUADRANT  III 

Which  data  systems  provide 
information  needed  to  measure 
performance 


Figure  8,1.  Four  Quadrant  Matrix 


A  detailed  procedure  for  defining  information  requirements  is 
given  in  King  and  Clealand  (1975) 

8.  3  Future  ’I'rends 

It  is  difficult  to  project  the  future  of  computer  based  MIS, 
however,  certain  trends  in  certain  elements  of  this  system  can 
he  identified,  and  could  be  summarized  as  follows  (Murdick  and 
Ross,  1975): 

-  The  changing  nature  of  MIS 

-  Real  time  and  time  sharing 

-  Information  Technology 

-  The  people  problem 

8.  1.1  The  Changing  Nature  of  MIS.  The  shift  away  from  hardware 
and  office  automation  into  improved  system  design  for  managerial 
use  will  continue,  where  the  objective  will  be  Improved  systems 
for  management  applications. 

8.  ).2_  Real  time  and  time  sharing.  D.espite  the  debate  whether 

r  ■  •  I 

management  requires  real  time  capabilities,  use  of  realtime  is 
going  to  accelerate,  mainly  because  the  improvement  in  computer 
communications  system.  This  will  enable  accessing  data  bases,  model 
building  and  query.  Systems  will  become  much  more  commonplace,  moving 
in  the  direction  of  decision  support  systems  (DSS) .  The  improvement 
in  computer  communications  systems  will  also  impinge  on  the  tendency 
to  use  centralized  data  bases  via  time  sharing. 

8.3.3  Information  Technology.  Some  improvement  in  hardware 
technology  and  use  are  going  to  take  place,  in  the  following 
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v 1 emen  t  s : 

-  data  communication 

-  data  storage  technology 

-  man  machine  interface,  where  improved  direct  interrogation 
of  the  computer  is  going  to  be  achieved 

-  input/output  devices,  which  are  the  current  bottleneck, 
will  be  improved  by  use  of  remote  terminals,  optical  data 
recognition,  voice  input,  automatic  copying  equipment,  and 
computerized  indexing  systems. 

-  EDP  technology  will  be  merged  with  telecommunications 
technology. 

-  Further  development  of  minicomputers  will  enable  their 
utilization  in  one  of  three  modes: 

-  "Stand  alone"  applications 

-  "Front  end  system" 

-  Data  concentrators 

However,  apparently  the  biggest  impact  is  going  to  come  from 
three  technologies  that  are  going  to  merge  into  what  might  be 
labeled  "Information  Processing."  These  three  technologies 
are  currently  known  as: 

-  distributed  systems 

-  data  base  systems 

-  word  processing 

Distributed  systems  make  use  of  teleprocessing  and  miniaturization 
of  computers  (minis  and  micros) .  Data  base  systems  make  use  of  high 
density  direct  access  storage  devices,  and  word  processing  depends 


i 
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ho.ivilv  on  nini.Uur  i.Mt  ion  of  cnniputi'rs  and  direct  .1  re  css  storage 
is  well  as  relatively  inexpensive  terminal  devices.  These 
technologies  are  an  outgrowth  of  existing  technologies . 

Of  particular  importance  to  MIS  is  the  impact  of  distributed 
systems,  as  discussed  later. 

8.3.4  The  People  Problem.  The  pace  at  which  future 
developments  occur  will  depend  on  management's  response  to  these 
trends.  For  properly  coping  with  those  trends,  training  and 
educaiton  of  people,  both  within  the  organization  and  outside, 
is  probably  the  best  answer. 

8.4  Centralization  versus  Decentralization  and  Distributed  Systems 

The  issue  of  centralization  is  a  classical  issue  in  the  study 
of  organizations,  and  MIS  has  not  avoided  this  issue  too.  Putting 
ii  in  the  context  of  the  various  "inquiry  systems"  defined  in  Chapter 
4^  a  I.eibnitian  inquirer,  leaning  towards  logic  and  internal 
consistency,  would  probably  require  a  centralized  information  system. 
A  Hegelian  inquirer,  leaning  towards  conflicting  representations, 
would  probably  require  some  form  of  decentralization. 

On  a  less  philosophical  level,  economies  of  scale  have  induced 
a  thrust  towards  centralized  data  processing  in  the  sixties  and 
early  seventies.  However,  due  to  the  recent  and  postulated 
technological  developments,  economies  of  scale  are  no  longer  a  major 
issue,  since  the  economics  of  large  systems  versus  multiple  small 
systems  is  balanced.  On  the  other  hand,  decentralization  enhances 
better  acceptance  of  computers  and  improved  service  to  the  user 
due  to  closer  control. 
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According  to  Davis  (1974)  six  main  strategies  can  be  considered 
to  combine  centralization  and  decentralization  of  information  processing 

systems : 

1.  Central  control  of  all  EDP  functions. 

2.  Central  advisory  function  with  all  the  information  processing 
development  and  operations  remaining  in  the  sub  units. 

3.  Central  control  of  hardware  and  software  operations  with 
decentralized  system  development  and  programming. 

■*.  Central  control  of  all  EDP  hardware,  operations  and 
programming,  with  only  systems  development  being 
decentralized. 

5.  Central  control  of  planning,  analysis  and  programming  with 
decentralization  of  hardware. 

6.  Distributed  computing  with  both  hardware  and  software  partly 
centralized  and  partly  decentralized. 

basically,  today,  the  centralization-decentralization  problem 
is  more  a  political  issue  rather  than  a  technological  problem.  A 
compromise,  responding  to  both  issues,  is  emerging  today  under 
the  heading  of  "Distributed  Systems."  The  technical  elements  of 
this  approach  were  defined  in  the  previous  section,  and  it  might 
be  worthwhile  to  examine  some  of  its  organizational  aspects. 

The  terra  "distributed  systems"  may  mean  different  things  to 
different  people.  To  some,  the  term  implies  distributed  files  or 
data  bases,  to  others,  distributed  CPUs,  and  both  still  to  others. 
Following  Walsh  (1978),  the  following  distinction  is  made: 


-  Distributed  processing  describes  an  orderly  fragmentation 

el  processing  among  two  or  more  computers  with  the  processing 
controlled  by  a  centrally  located  computer  commonly  known  as 
a  host. 

-  Distributed  data  bases  indicates  an  orderly  fragmentation  ot 
data  bases  or  files  among  the  peripheral  storage  devices  of 
one  or  more  computer  configurations 

-  Distributed  systems  describes  the  hardware  and/or  software 
configuration  of  a  system  in  which  distribution  processing 
takes  place. 

Thus,  distributed  systems  represent  technology's  contribution 
towards  resolving  the  old  "centralization  versus  decentralization 
issue.  It  is  this  kind  of  system  that  enables  implementation  of 
any  type  of  the  six  centralization/decentralization  strategies 
described  above.  As  such,  distributed  systems  is  a  contemporary 
phenomenon  that  seems  to  be  here  to  stay. 

b. 5  MIS  Design  Process:  Comments 

The  importance  of  requirement  planning  for  the  MIS  design 
process  was  emphasized  already  in  Section  8.2  The  design  process 
affects  both  MIS  effectiveness  and  its  evaluation,  especially  in 
the  sense  of  embedding  during  the  design,  evaluation  components 
in  the  system.  Following  therefore,  are  some  comments  related  to 
the  design  process. 

Effective  system  design  cannot  take  place  in  a  managerial 
vacuum,  therefore,  management  interest,  involvement  and  support 
is  required  at  all  levels.  Specifically,  it  is  important  to  ascertain 
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that  the  following  elements  exist: 


-  User's  participation  -  not  only  in  defining  the  requirements, 
hut  in  the  design  process  itself 

-  Top  management  support 

-  Insure  that  the  designed  system  maintains  the  information 
attributes.  Thus,  for  example,  if  attributes  such  as 
timeliness,  relevance,  flexibility  etc.  are  missing, 

the  chances  of  this  MIS  being  effective  are  minimal. 

Especially  important  as  part  of  the  MIS  design  process  is  to 
design  a  "maintenance"  capability,  that  could  perform,  once  the 
system  is  in  operation,  a  function  of  product  enhancement  (providing 
new  functional  capabilities),  product  improvement  (i.e.,  increasing 
its  reliability  or  supportability)  and  the  correcting  of  anomalous 
behavior  due  to  design  oversights. 

One  consequence  if  this  aspect  of  MIS  is  that  the  system  is 
usually  at  some  level  of  continuing  development;  therefore,  it 
appears  to  be  never  completed  to  any  observer  who  believes  the 
myth  that  turnkey  MIS  can  be  produced. 

The  stage  is  set  now  to  identify  research  needs,  which  is  done 
in  the  next  chapter. 
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9.  RESEARCH  NEEDS 


It  is  possible  now,  based  upon  the  discussion  throughout 
this  research,  to  identify  research  needs  in  the  general  area  of 
MIS  software  e f f ec t iveness .  The  research  needs  were  classified 
in  three  groups,  as  follows: 

-  Evaluation  Methodology 

-  Military  MIS 

-  Impact  of  future  trends 

9.1  Evaluation  Methodology 

In  Chapter  6,  a  thorough  analysis  of  the  "state  of  the  art" 
of  evaluation  techniques  was  performed,  and  summarized  in  Table 
7-4,  in  relation  to  the  management  information  metrics  that  was 
developed  in  Chapter  4.  The  general  conclusion  that  can  be 
drawn  is  that  there  is  no  one  satisfactory  approach  that  can 
measure  and  evaluate  MIS  software  effectiveness  from  the  user's 
Recalling  the  two  stage  process  of  evaluation  and 
measurement  discussed  in  5.3,  this  state  of  affairs  can  be 
attributed  to  two  problems'. 

-  deficiencies  in  establishing  the  theoretical  metric  (the 
"volt") 

-  shortcomings  of  the  measuring  devices  (the  "voltmeter") 

It  should  also  be  clear,  like  in  many  other  instances,  that 

there  can  be  no  single  value  that  could  assess  the  MIS  effectiveness. 
Even  if  general  attributes  like  "user  satisfaction"  or  "relevance" 
could  be  measured  without  any  ambiguity,  they  still  do  not  cover 


all  aspects  oi  effectiveness. 


Those  general  observations  have  to  be  broken  down  into  their 
components  in  order  to  identify  more  specific  research  areas.  Those 
specific  areas  are  presented  now. 

9.1.1  Techniques  Research.  In  Table  7-4,  the  extent  to  which 
available  techniques  can  evaluate  management  information  attributes 
was  displayed.  Additional  information  can  be  extracted  from  this 
table  by  performing  the  following  analysis:  the  various  techniques 
were  lumped  together  into  their  major  four  groups  -  economic, 
behavioral,  other  and  management  science.  Also,  the  various 
attributes  were  lumped  together  into  the  three  major  groups  - 
management,  information  and  system.  Now,  a  count  has  been  made  of 
the  number  of  possible  applications  of  a  certain  technique  group  to 
a  certain  attribute  group,  regardless  of  the  ranking  of  this 
application  or  the  quality  of  the  technique.  The  resulting  matrix 
is  shown  in  Table  9-1. 


Table  9-w-l  Count  of  Evaluation  Techniques  Applications 


Mgt. 

Inf. 

Sys . 

Economic 

4 

1 

1 

Behavioral 

7 

1 

- 

Other 

3 

5 

3 

Mgt.  Science 

2 

- 

- 

It  is  obvious  that  the  most  attention,  by  implication,  has  been 
given  to  the  "management"  group  of  metrics,  and  not  surprisingly 
so,  however,  without  too  much  success,  as  most  of  those  techniques 


art'  t'f  the  behnvior.il  type  that  was  already  identified  as  being  deficient. 


On  the  other  hand,  the  other  two  groups  will  require  either  the 
objective  or  mixed  evaluation  approach  (Table  7-3),  possibly  cheaper 
techniques  which  should  be  much  simpler  and  less  problematic  in  their 
application.  However,  those  two  groups  received  less  attention.  The 
appealing  research  needs  emerging  from  this  analysis  are  as  follows: 

-  expansion  and  development  of  the  measurement  and  evaluation 
techniques  for  the  "information"  and  "system"  group  of 
attributes,  emphasizing  objectivity  of  measurement  and  low 
cost . 

-  research  into  the  possibility  of  using  the  above  groups  of 
attributes  as  a  "surrogate"  measure  to  evaluate  the  management 
group  of  attributes,  especially  those  that  require  a  pure 
behavioral  approach.  Should  this  be  possible,  then  the 
attributes  of  the  "management"  group  could  be  evaluated  by 

a  cheaper  and  simpler  method,  thus  avoiding  the  problems  of 
the  behavioral  techniques,  which  apparently  will  not 
disappear  anytime  soon. 

h.1,2  Management  Information  Dimensionality.  It  was  pointed  out 
before,  that  there  can  be  no  single  value  that  will  measure  MIS 
effectiveness.  Due  to  the  multi  dimensional  nature  of  the  management 
information  metrics,  evaluating  effectiveness  becomes  to  be  a  complicated 
issue.  Suppose,  for  example,  that  a  numerical  value  could  be  assigned 
to  each  one  of  the  attributes.  The  basic  evaluation  issue  is  not 
resolved,  as  measuring  each  attribute  does  not  say  a  thing  yet  about 
the  effectiveness  of  the  system.  This  points  out  the  following  specific 
research  needs: 
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-  setting  standards  for  "good"  values  of  attribute  measurement 
(those  standards  may  vary  from  system  to  system) 

-  methodology  for  evaluating  the  effectiveness  of  the  system 
based  upon  measurement  of  the  attributes.  This  amounts  to 
giving  the  system  effectiveness  meaning  based  upon  measuring 
the  elements,  where  some  values  are  "good,"  and  some  "bad." 

An  analogy  from  a  different  field;  Volt  and  ampere  are 
measurements  of  attributes.  Watt  is  an  evaluation  of 
effectiveness. 

9.1.3  Impact  of  the  Decision  Environment.  In  previous  chapters, 
different  aspects  of  the  decision  environment  were  discussed,  such 
as:  different  types  of  decision  making  processes,  different  types 

of  decisions  according  to  the  organizational  hierarchy  (i.e., 
strategic,  tactical,  operational),  different  contents  of  decisions 
(i.e.,  planning,  staffing,  controlling,  etc.)  -  in  short,  a  varied 
decision  environment.  It  seems  reasonable  then  that  effectiveness 
evaluation  should  consider  the  decision  environment.  This  suggests 
the  following  research  issue: 

-  Evaluating  MIS  effectiveness  for  different  organizational 
levels,  say  three:  Top,  middle  and  operations  management. 

9.1.4  Design  Phase  Evaluation.  By  and  large,  it  can  be  seen 
from  Table  7-4  that  the  effectiveness  evaluation  during  the  operation 
phase  is  better  covered  than  that  during  the  design  phase.  The 
importance  of  effectiveness  evaluation  during  the  design  phase  stems 
from  two  reasons: 

-  investment  justification 

-  assessment  of  the  system  effectiveness  after  implementation 


Typically,  the  economic  approaches  seem  to  hold  the  highest 
"hope"  for  this  phase,  however,  it  was  pointed  out  that  potential 
savings  or  benefits  are  hard  to  measure,  and  apparently,  further 
progress  in  this  direction  is  going  to  be  difficult.  New  approaches 
should  be  tried  -  specifically,  the  following  research  is  suggested: 

-  use  of  simulation  for  design  phase  evaluation 

-  use  of  real  world  experiment,  i.e.  a  pilot  project. 

9.1.5  Methodology  for  Continuous  Review  and  Evaluation.  MIS 
"operates"  within  a  dynamic  environment,  where  often  times  both  the  nature 
of  the  operation  and  the  decision  maker  are  changing.  This  implies  that 
effectiveness  may  not  be,  once  established,  a  constant  feature  of  the 
system.  Furthermore,  a  drop  in  effectiveness  may  require  some 
"maintenance"  activities.  All  this  points  out  that  a  methodology 
for  continuous  evaluation  has  to  be  developed.  It  seems  that  the 
continuous  evaluation  is  going  to  be  less  comprehensive  -  in  terms  of 
attribute  coverage  -  than  the  "discrete"  evaluation,  and  have  the 
fill  lowing  feature. 

-  low  cost  and  not  time  consuming 

-  composed  of  user  feedback  and  objective  measurement,  preferably 
automated  (such  as  volume  measurement  etc.  -  note  the  idea 

of  traces  in  Chapter  6.) 

9.2  Military  MIS 

The  research  needs  included  in  this  section  imply  that  the  specific 
flavor  of  a  military  system  should  be  taken  into  account. 

9.2.1  Requirement  Planning.  Requirement  planning  is  a  recognized 
process  today,  and  was  identified  as  one  of  the  key  elements  in  MIS 
design  and  am  important  element  in  MIS  evaluation.  The  existing 
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approaches  need  some  development,  and  research  is  required  in  two  areas 


-  a  methodology  for  a  "system  wide"  requirement  planning,  where 

the  needs  identified  are  used  to  assess  the  MIS  effectiveness  during 
the  design  phase. 

-  means  of  measuring  the  needs  fulfillment  during  the  operation 
phase,  as  a  measure  of  the  system  effectiveness. 

9.2.2  Management  Information  Attributes  Ranking.  The  management 
information  metrics  developed  includes  a  high  number  of  attributes, 
some  probably  more  important,  some  less  for  a  specific  system. 

This  points  out  to  a  research  need  for  ranking  the  attributes, 
serving  two  goals: 

-  possible  weighting  of  the  attributes  for  effectiveness  evaluation 

-  where  measurement  methods  do  not  exist,  then  the  ranking  will 
identify  those  attributes  for  which  the  research  effort  should 
be  directed  first. 

9.3  Impact  of  Future  Trends 

Future  trends  may  impact  the  effectiveness  issue  in  two  ways: 

-  change  the  effectiveness  of  MIS  by  "scoring"  a  higher  values 
on  some  of  the  attributes  measured. 

-  enable  better  measurement  of  some  information  attributes 
Specific  research  needs  in  this  area  are: 

-  distributed  systems 

-  interactive  systems 

In  both  cases,  the  specific  issues  to  be  investigated  relate 
to  the  comments  made  above,  namely: 

-  possible  increase  in  MIS  effectiveness  due  to  use  of  those 
systems 

-  embedding  of  measurement  tools  in  the  system  itself. 


-68- 


9.4  Research  Clusters 

The  research  needs  identified  were  defined  as  separate  issues. 
However,  there  are  research  topics  more  related  to  each  other  than 
others.  Thus,  a  grouping  of  related  topics  has  been  done,  yielding 
"research  clusters,"  as  follows: 

Measurement  Cluster 

-  Expansion  and  development  of  the  measurement  and  evaluation 
techniques  for  the  "information"  and  "system"  group  of 
attributes 

-  Use  of  "infomration"  and  "system"  group  of  attributes  as  a 
surrogate  measure  for  evaluating  the  "management"  group 

-  Setting  standards  for  attribute  measurements 

-  Attribute  ranking 

Kf fectiveness  Cluster 

-  Methodology  for  evaluating  the  effectiveness  of  the  system 
based  upon  measuring  the  attributes 

-  Evaluation  of  effectiveness  for  different  organizational  levels. 

-  Methodology  for  "system  wide"  requirement  planning,  where  the 
needs  identified  are  used  to  evaluate  effectiveness 

-  means  of  measuring  need  fulfillment  as  an  indication  of 
effectiveness . 

Design  Phase  Cluster 

-  Use  of  simulation  for  design  phase  evaluation 

-  Use  of  real  word  experiments  -  a  pilot  project 

Future  Trends  Cluster 

-  Methodology  for  continuous  review  and  evaluation 
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10.  CONCLUSIONS  AND  RECOMMENDATIONS:  MIS  EFFECTIVENESS  PROJECT  - 
GENERAL  GUIDELINES 

Now  that  the  various  research  needs  have  been  identified,  it  is 
possible  to  suggest  the  approach  for  an  MIS  effectiveness  project 
within  the  Army  system.  The  approach  proposed  has  a  modular  structure, 
based  upon  the  four  research  clusters  defined  in  the  previous  chapter. 


Figure  10-1:  Modules  of  MIS  Software  Effectiveness  Project 

Thus,  the  four  modules  are: 

-  Measurement  module 

-  Effectiveness  module 

-  Design  phase  module 

-  Future  trends  and  effectiveness  module 

The  modular  approach  gives  the  flexibility  of  performing  the 
whole  project  or  working  on  each  module  separately.  It  also  enables 
splitting  the  project  between  different  researchers  and  geographical 
locations.  If  a  sequential  approach  is  to  be  used.  It  is  recommended 
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that  tho  modules  be  implemented  in  the  sequences  presented,  i.e. 
from  left  to  right. 

The  basic  contents  of  each  module  was  described  in  Chapter  9. 

Some  additional  comments  are  added  here. 

10. l_  Measurement  Module 

This  module  contains  four  major  research  areas,  briefly 
summarized  as: 

-  measurement  techniques 

-  surrogate  measures 

-  standards 

-  attribute  ranking 

Probably  the  area  to  start  with  is  attribute  ranking.  Standards 
can  be  based  upon  the  "service  level"  desired,  historical  records 
if  available,  and  could  serve  as  a  monitoring  tool  too.  Investigation 
of  the  possibility  of  surrogate  measures  is  apparently  the  most 
difficult  research  task  in  this  module,  and  probably  the  most 
important . 

10.2  Effectiveness  Module 

This  module  is  basically  concerned  with  the  "heart"  of  the 
effectiveness  issue,  i.e.,  research  into  evaluation  of  effectiveness, 
once  measurements  are  available.  Definitely,  part  of  the  activities 
in  this  module  depend  on  the  results  obtained  in  the  previous  module. 
Recapping^  the  research  topics  in  this  module  are: 

-  effectiveness  evaluation  methodology  based  upon  attribute 
measurements 
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-  of fectiveness  and  organizational  hierarchy 

-  methodology  for  "system  mode"  requirement  planning 

The  methodology  for  system  wide  requirement  planning  has  to 
consider  the  specifics  of  the  army  environment. 

It  is  recommended  that  as  part  of  this  module,  this  methodology 
be  applied,  where  the  key  factors  are  user's  participation  and 
management  support.  Furthermore,  it  is  recommended  that  a  comparison 
among  the  requirements  of  all  command  levels  and  STAMMIS  be  performed 
with  the  objective  of  identifying  what  may  be  required  to  adapt  the 
current  system  to  the  defined  needs. 

10.3  Design  Phase  Module 

Both  research  areas  suggested  -  simulation  and  pilot  project  - 
were  used  in  other  environments  for  evaluating  effectiveness  during 
the  design  phase.  However,  further  research  is  required  to  adapt  them 
to  the  army  environments  as  both  approaches  -  by  their  nature  -  have 
to  be  "tailor  made"  for  the  system  analyzed. 

10.4  Future  Trends  Module 

This  module  could  be  started  only  after  some  results  were  obtained 
in  the  first  two  modules.  The  research  in  continuous  review  methodology 
is  aimed  at  making  the  MIS  effectiveness  issue  a  managerial  function 
that  has  to  be  continuously  monitored.  This  could  alleviate  some  of 
the  current  problems  of  the  existing  system. 

Distributed  systems  and  interactive  systems  are  not  a  new  phenomena, 
however,  their  impact  on  effectiveness  has  to  be  researched,  and  the 
possibility  of  utilizing  those  systems  for  attribute  measurement  has  to 
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be  investigated. 


10.4  Conclusions  and  Recommendations 

The  major  findings  of  this  research  are  as  follows: 

-  there  is  no  one  satisfactory  approach  that  can  measure  and 
evaluate  MIS  software  effectiveness 

-  the  management  group  of  attributes  of  the  management  information 
metrics  received  more  attention  without  very  much  success,  as 
the  techniques  used  are  of  the  behavioral  type  -  most  of  them 
based  on  questionnaires  -  that  have  not  matured  yet,  and  no 
drastic  change  is  anticipated  in  the  near  future. 

-  the  "information"  and  "system"  group  of  attributes  of  the 
management  information  metrics  has  received  less  attention, 
however,  hold  more  potential  for  evaluating  effectiveness  because 
of  the  more  objective  measurement  possible  within  this  group. 
Also,  possibly  the  attributes  here  could  serve  as  a  surrogate 
measure  of  the  management  group  of  attributes,  thus  alleviating 
some  of  the  evaluation  problems  there. 

-  no  methodology  exists  for  evaluating  the  effectiveness  of  the 
system  based  upon  measurement  of  the  attributes. 

-  available  techniques  for  evaluating  MIS  effectiveness 
during  the  design  phase  do  not  enable  economic  justification. 

-  the  requirement  planning  process  does  not  overtly  consider 
the  effectiveness  evaluation  problem. 

It  is  recommended  that  an  MIS  software  effectiveness  project  be 
designed  and  implemented,  following  the  modular  structure  presented 
above.  The  implementation  policy  could  use  a  parallel  approach  - 


implementing  all  four  modules  at  the  same  time,  or  a  sequential  approach 
one  module  at  a  time,  where  the  priorities  are 

-  measurement  module 

-  effectiveness  module 

-  design  phase  module 

-  future  trends  module 

The  sequential  approach  is  recommended.  Furthermore,  it  is 
recommended  that  the  first  two  modules  be  implemented  concurrently. 
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